Welcome to papermill¶
Papermill is a tool for parameterizing and executing Jupyter Notebooks.
Papermill lets you:
- parameterize notebooks
- execute notebooks
This opens up new opportunities for how notebooks can be used. For example:
- Perhaps you have a financial report that you wish to run with different values on the first or last day of a month or at the beginning or end of the year, using parameters makes this task easier.
- Do you want to run a notebook and depending on its results, choose a particular notebook to run next? You can now programmatically execute a workflow without having to copy and paste from notebook to notebook manually.
Python Version Support¶
This library currently supports python 3.6+ versions. As minor python versions are officially sunset by the python org papermill will similarly drop support in the future.
Documentation¶
These pages guide you through the installation and usage of papermill.
Installation¶
Change Log¶
2.1.3¶
- Removed jupyter_client dependency in requirements to avoid confusing pip on the actual version requirements.
- Parameterized commenting so that once can pass a
comment
argument to assign the comment string in injected cells.
2.1.2¶
- Expand Usage Docs for JupyterLab
- Support
nan
andinf
in Python translator - Added fix for required async loop registration in python 38 on windows
2.1.1¶
- DeadKernelExceptions, usually from OOM, now exit with a status code of 138 from the CLI.
- Error cell at the top of failed notebook has been made better. It now also has a link to an injected cell where the error occurred.
- Updated a deprecated function to the new function name for nbclient dependency.
- Some development and documentation updates / fixes have also been made by a few different contributions (thank you!).
2.1.0¶
- Support for python 3.5 has been dropped. Upstream library changes for async were causing process deadlocks with await commands. End-of-life is later this year for 3.5 anyway so we decided to also drop support here.
- Error cells injected at the top of failed notebooks look nicer now as markdown.
2.0.0¶
Papermill 2.0 has a number of awesome features from many different contributors. We used the major version change mostly to signify the change to Python 3 only, but we also allowed for PRs which has small interaction changes to also be made. No major functionality should change with this release, but many minor improvements might impact specific execution patterns. We’ll keep an eye on issues and post bug fixes ASAP if any of these cause larger unexpected issues.
Features¶
- Papermill is now Python 3.5+ only!
- nbconvert is no longer a dependency of papermill, instead the smaller and newly released nbclient is now the execution dependency.
- Support added for parameterizing C# kernels
- Support added for parameterizing F# kernels
sys.exit(0)
now respected by papermill- Python parameters are now black formatted (in python versions >= 3.6)
- Notebook documents are saved periodically now rather than solely on cell completion.
- A cell
--execute-timeout
option was added. - HDFS io support added with
hdfs://
scheme (withpapermill[hdfs]
install).
Fixes¶
- Fixed metadata writing on markdown and raw cells to follow v4.4 schema correctly
- Azure Blob Storage support fixed for newer blob storage.
azure-storage-blob >= 12.1.0
is now supported, older version support was dropped. - IOPub timeouts now raise an exception instead of a warning.
Interaction Changes (more details)¶
- nbconvert dependency has been replaced with nbclient. This means the default engine is now
nbclient
rather thannbconvert
and the NBConvertEngine class no longer exists. This may mean extensions that extended this class will need to be updated slightly to the new class. sys.exit(0)
in python kernels now transfers exit code to papermill, meaning papermill will gracefully stop the notebook execution and not raise an exception to the user.sys.exit(1)
or other exceptions still raise as expected and change the status code from 0 for the papermill process.- When generating parameters for python (when running on 3.6+) the parameters will be printed more cleanly with a pass of black before injecting into the notebook.
- Older Azure Blob Storage support was dropped:
azure-storage-blob < 12.1.0
- The
--autosave-cell-every
option now controls the minimum time between notebook saves during cell execution. This time will exponentially backoff if it takes more than 25% of the autosave-cell-every value. Setting--autosave-cell-every
to0
disabled this feature. - The
--execute-timeout
option can be set to enable a per-cell execution timeout limit. - IOPub timeouts used to only warn and attempt to continue execution. This can be triggered by printing ‘0’ in a wide for-loop without any sleeps. The side-effect of best-effort execution was that outputs and failures could be lost in the IOPub timeout event and notebooks would “succeed” when they were actually failing. We chose to change this pattern from a warning to an error for papermill. To fix the issue when it occurs you need to delay the number of print or display messages per second being produced in your notebooks.
1.2.1¶
- Importing papermill no longer manipulates
yaml.SafeLoader
globally - Parameters with leading
_
now have prefix_
stripped before passing to R kernels - A few documentation typos were fixed
1.2.0¶
- Parameters lists passing feature from 1.1.0 was removed due to cli api issues it caused.
- Piping papermill into nbconvert no longer triggers an encoding error on Python 2
- Added
BOTO3_ENDPOINT_URL
environment variable to override boto session url - stdout / stderr can now be streamed to a file via
--stdout-file /dev/stdout
and--stderr-file /dev/stderr
. - The CLI option
--not-report-mode
is now--no-report-mode
- GCFS connectors should now retry under all conditions that the upstream library defines as retryable. Papermill now uses the is_retryable method from the upstream dependency.
1.1.0 (This version should be avoided for several known issues fixed in 1.2.0)¶
- Read content from stdin/to stdout when the path is
-
or a pipe. This allows for<generate input>... | papermill | ...<process output>
, withpapermill - -
being implied by the pipes. - The built-in
ADLHandler
for Azure Pipelines should now work properly again. - Many documentation improvements
- IPython is now lazily imported only when progress bars are needed.
- A MATLAB translator is now available for parameters being passed to MATLAB notebooks.
- Parameters lists can more easily be passed to the command line via:
-p name value1 value2 3 ...
which results in adding to notebooks a parameter list assignmentname = ["value1", "value2", 3]
.
1.0.1¶
- Cleaned up some dependency and build issues around pip 19 and pandas
execute_notebook
can now take notebook as strings instead of only as a pathkwargs
are now passed through the default engine to nbconvert’s wrapper class- Passing dates through yaml as parameters will no longer raise an exception (i.e.
-y "a_date: 2019-01-01"
without having to quote ala-y "a_date: '2019-01-01'"
)
1.0.0¶
We made it to our 1.0 milestone goals! The largest change here is removal of record
, Notebook
, and NotebookCollection
abstractions which are now living in scrapbook and requirement of nbconvert 5.5 as a dependency.
- Input and output paths can now reference input parameters.
my_nb_{nb_type}.ipynb out_{nb_type}.ipynb -p nb_type test
will substitute values into the paths passed in with python format application patterns. read_notebook
,read_notebooks
,record
, anddisplay
api functions are now removed.- [upstream] ipywidgets are now supported. See nbconvert docs for details.
- [upstream] notebook executions which run out of memory no longer hang indefinitely when the kernel dies.
0.19.1¶
- Added a warning when no
parameter
tag is present but parameters are being passed - Replaced
retry
withtenacity
to help with conda builds and to use a non-abandoned library
0.19.0¶
DEPRECATION CHANGE The record, read_notebook, and read_notebooks functions are now officially deprecated and will be removed in papermill 1.0.
- scrapbook functionality is now deprecated
- gcsfs support is expanded to cover recent releases
0.18.1¶
Fixes¶
- azure missing environment variable now has a better error message and only fails lazily
- gcs connector now has a backoff to respect service rate limits
0.18.0¶
INSTALL CHANGE The iorw extensions now use optional dependencies. This means that installation for s3, azure, and gcs connectors are added via:
pip install papermill[s3,azure,gcs]
or for all dependencies
pip install papermill[all]
Features¶
- Optional IO extensions are now separated into different dependencies.
- Added gs:// optional dependency for google cloud storage support.
- null json fields in parmaeters now translate correctly to equivilent fields in each supported language
Fixes¶
- tqdm dependencies are pinned to fetch a minimum version for auto tqdm
Dev Improvements¶
- Releases and versioning patterns were made easier
- Tox is now used to capture all test and build requirements
0.17.0¶
Features¶
- Log level can now be set with
--log-level
- The working directory of papermill can be set with the
--cwd
option. This will set the executing context of the kernel but not impact input/output paths.papermill --cwd foo bar/input_nb.ipynb bar/output_nb.ipynb
would make the notebook able to reference files in thefoo
directoy without../foo
but still save the output notebook in thebar
directory. - Tox has been added for testing papermill. This makes it easier to catch linting and manifest issues without waiting for a failed Travis build.
Fixes¶
- Fixed warnings for reading non-ipynb files
- Fixed
--report-mode
with parameters (and made it more compatible with JupyterLab) - Papermill execution progress bars now render within a notebook correctly after importing seaborn
- The
--prepare-only
option no longer requires that kernels be installed locally (you can parameterize a notebook without knowing how to execute it) - Azure IO adapter now correctly prefixes paths with the
adl://
scheme - Tests on OSX should pass again
Docs¶
- Install doc improvements
- Guide links are updated in the README
- Test docs updated for tox usage
0.16.2¶
- Injected parameter cells now respect
--report-mode
- Logging level is only set for logger through CLI commands
- Output and input paths can be automatically passed to notebooks with the
--inject-paths
option - Entrypoints have been added for registration of new
papermill.io
andpapermill.engine
plugins via setup files
0.16.1¶
- Fixed issue with azure blob io operations
0.16.0¶
- Added engines abstraction and command line argument
- Moved some nbconvert wrappers out of papermill
- Added Azure blob storage support
- Fixed botocore upgrade comptability issue (all version of boto now supported again)
- Removed whitelisted environment variable assignment
0.15.1¶
- Added support for Julia kernels
- Many improvements to README.md and documentation
- nbconvert dependency pinned to >= 5.3
- Improved error handling for missing directories
- Warnings added when an unexpected file extension is used
- Papermill version is visible to the CLI
- More messages us logging module now (and can be filtered accordingly)
- Binder link from README was greatly improved to demostrate papermill features
0.15.0¶
- Moved translator functions into registry
- Added development guide to help new contributors
- Travis, coverage, linting, and doc improvements
- Added support for Azure data lake sources
- Added python 3.7 testing
0.14.2¶
- Added output flushing for log-output option
0.14.1¶
- Upgraded executor to stream outputs during execution
- Fixed UTF-8 encoding issues for windows machines
- Added black code formatter rules (experimental)
- Contributors document added
- Added report-mode option for hiding inputs
0.13.4 (no code changes)¶
- Release manifest fix
0.13.3¶
- Fixed scala int vs long assignment
0.13.2¶
- Pip 10 fixes
0.13.1¶
- iPython pin to circumvent upstream issue
0.13.0¶
- Added prepare-only flag for parameterizing without processing a notebook
- Fixed cell number display on failed output notebooks
- Added scala language support
0.12.6¶
- Changed CLI outputs from papermill messaging to stderr
- Changed IOResolvers to perseve ordering of definition in resolving paths
0.12.5¶
- Set click disable_unicode_literals_warning=True to disable unicode literals
0.12.4¶
- Added universal wheel support
- Test coverage for s3 improved
0.12.3¶
- Added start timeout option for slow booting kernels
0.12.2¶
- Added options around tqdm
- Fixed an S3 decoding issue
0.12.1¶
- ip_display improvements
- Docstring improvements
0.12.0¶
- Added type preservation for r and python parameters
- Massive test coverage improvements
- Codebase style pass
Usage¶
For an interactive example that demonstrates the usage of papermill, click the Binder link below:
Using papermill¶
The general workflow when using papermill is parameterizing a notebook, executing it, as well as storing the results. In addition to operating on a single notebook, papermill also works on a collection of notebooks.
Parameterize¶
See also
Generally, the first workflow step when using papermill is to parameterize the notebook.
To do this, tag notebook cells with parameters
. These parameters
are
later used when the notebook is executed or run.
Designate parameters for a cell¶
To parameterize a notebook, designate a cell with the tag parameters
.
If using the Jupyter Notebook interface
- Activate the tagging toolbar by navigating to
View
,Cell Toolbar
, and thenTags
- Enter
parameters
into a textbox at the top right of a cell - Click
Add tag
If using the JupyterLab interface
- Select the cell to parameterize
- Click the property inspector on the left side (double gear icon)
- Type “parameters” in the “Add Tag” box and hit “Enter”.
If using JupyterLab < 2.0, consider using the jupyterlab-celltags extension.
- Select the cell to parameterize
- Click the cell inspector (wrench icon)
- Type “parameters” in the “Add Tag” box and hit “Enter”.
If the extension is not installed, you can manually add the tag by editing the Cell Metadata field in the cell inspector by adding “tags”:[“parameters”].
Learn more about the jupyter notebook format and metadata fields here.
How parameters work¶
The parameters
cell is assumed to specify default values which may be
overridden by values specified at execution time.
- papermill inserts a new cell tagged
injected-parameters
immediately after theparameters
cell injected-parameters
contains only the overridden parameters- subsequent cells are treated as normal cells, even if also tagged
parameters
- if no cell is tagged
parameters
, theinjected-parameters
cell is inserted at the top of the notebook
One caveat is that a parameters
cell may not behave intuitively with
inter-dependent parameters. Consider a notebook note.ipynb
with two cells:
#parameters
a = 1
twice = a * 2
print("a =", a, "and twice =", twice)
when executed with papermill note.ipynb -p a 9
, the output will be
a = 9 and twice = 2
(not twice = 18
).
Execute¶
The two ways to execute the notebook with parameters are: (1) through the Python API and (2) through the command line interface.
Execute via the Python API¶
The execute_notebook function can be called to execute an input notebook when passed a dictionary of parameters:
execute_notebook(<input notebook>, <output notebook>, <dictionary of parameters>)
import papermill as pm
pm.execute_notebook(
'path/to/input.ipynb',
'path/to/output.ipynb',
parameters=dict(alpha=0.6, ratio=0.1)
)
Execute via CLI¶
To execute a notebook using the CLI, enter the papermill
command in the
terminal with the input notebook, location for output notebook, and options.
See also
Here’s an example of a local notebook being executed and output to an Amazon S3 account:
$ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
In the above example, two parameters are set: alpha
and l1_ratio
using -p
(--parameters
also works).
Parameter values that look like booleans or numbers will be interpreted as such.
Here are the different ways users may set parameters:
Using -r
or --parameters_raw
, users can set parameters one by one. However, unlike -p
, the parameter will
remain a string, even if it may be interpreted as a number or boolean.
$ papermill local/input.ipynb s3://bkt/output.ipynb -r version 1.0
Using -f
or --parameters_file
, users can provide a YAML file from which parameter values should be read.
$ papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml
Using -y
or --parameters_yaml
, users can directly provide a YAML string containing parameter values.
$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
- 0.0
- 1.0
- 2.0
- 3.0
linear_function:
slope: 3.0
intercept: 1.0"
Using -b
or --parameters_base64
, users can provide a YAML string, base64-encoded, containing parameter values.
$ papermill local/input.ipynb s3://bkt/output.ipynb -b YWxwaGE6IDAuNgpsMV9yYXRpbzogMC4xCg==
When using YAML to pass arguments, through -y
, -b
or -f
, parameter values can be arrays or dictionaries:
$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
- 0.0
- 1.0
- 2.0
- 3.0
linear_function:
slope: 3.0
intercept: 1.0"
If you use multiple AWS accounts and are accessing S3 files, you can [configure your AWS credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html), to specify which account to use by setting the AWS_PROFILE environment variable at the command-line. For example:
$ AWS_PROFILE=dev_account papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
A similar pattern may be needed for other types of remote storage accounts.
Store¶
See also
Papermill can store notebooks in a number of locations including AWS S3, Azure data blobs, and Azure data lakes.
The modular architecture of papermill allows new data stores to be added over time.
See also:
Command Line Interface¶
papermill may be executed from the terminal. The following are the command options:
Usage: papermill [OPTIONS] NOTEBOOK_PATH OUTPUT_PATH
This utility executes a single notebook in a subprocess.
Papermill takes a source notebook, applies parameters to the source
notebook, executes the notebook with the specified kernel, and saves the
output in the destination notebook.
The NOTEBOOK_PATH and OUTPUT_PATH can now be replaced by `-` representing
stdout and stderr, or by the presence of pipe inputs / outputs. Meaning
that
`<generate input>... | papermill | ...<process output>`
with `papermill - -` being implied by the pipes will read a notebook from
stdin and write it out to stdout.
Options:
-p, --parameters TEXT... Parameters to pass to the parameters cell.
-r, --parameters_raw TEXT... Parameters to be read as raw string.
-f, --parameters_file TEXT Path to YAML file containing parameters.
-y, --parameters_yaml TEXT YAML string to be used as parameters.
-b, --parameters_base64 TEXT Base64 encoded YAML string as parameters.
--inject-input-path Insert the path of the input notebook as
PAPERMILL_INPUT_PATH as a notebook
parameter.
--inject-output-path Insert the path of the output notebook as
PAPERMILL_OUTPUT_PATH as a notebook
parameter.
--inject-paths Insert the paths of input/output notebooks
as
PAPERMILL_INPUT_PATH/PAPERMILL_OUTPUT_PATH
as notebook parameters.
--engine TEXT The execution engine name to use in
evaluating the notebook.
--request-save-on-cell-execute / --no-request-save-on-cell-execute
Request save notebook after each cell
execution
--prepare-only / --prepare-execute
Flag for outputting the notebook without
execution, but with parameters applied.
-k, --kernel TEXT Name of kernel to run.
--cwd TEXT Working directory to run notebook in.
--progress-bar / --no-progress-bar
Flag for turning on the progress bar.
--log-output / --no-log-output Flag for writing notebook output to the
configured logger.
--stdout-file FILENAME File to write notebook stdout output to.
--stderr-file FILENAME File to write notebook stderr output to.
--log-level [NOTSET|DEBUG|INFO|WARNING|ERROR|CRITICAL]
Set log level
--start-timeout INTEGER Time in seconds to wait for kernel to start.
--execution-timeout INTEGER Time in seconds to wait for each cell before
failing execution (default: forever)
--report-mode / --no-report-mode
Flag for hiding input.
--version Flag for displaying the version.
-h, --help Show this message and exit.
Extending papermill¶
Papermill provides some interfaces with external services out of the box. However, you may find that you would like papermill to do more than it currently does. You could contribute to the papermill project yourself (see Extending papermill by contributing to it). However, an easier method might be to extend papermill using entry points.
In general, when you run a notebook with papermill, the following happens:
- The notebook file is read in
- The file content is converted to a notebook python object
- The notebook is executed
- The notebook is written to a file
Through entry points, you can write your own tools to handle steps 1, 3, and 4. If you find that there’s more you want to contribute to papermill, consider developing papermill itself.
Extending papermill through entry points¶
The python packaging documentation describes entry points as:
Entry points are a mechanism for an installed distribution to advertise components it provides to be discovered and used by other code. For example:
Distributions can specify console_scripts entry points, each referring to a function. When pip (or another console_scripts aware installer) installs the distribution, it will create a command-line wrapper for each entry point.
Applications can use entry points to load plugins; e.g. Pygments (a syntax highlighting tool) can use additional lexers and styles from separately installed packages. For more about this, see Creating and discovering plugins.
When running, papermill looks for entry points that implement input / output (I/O) handlers, and execution handlers.
Virtually the first thing that happens when papermill is used is that the input notebook is read in. This is managed by I/O handlers, which allow papermill to access not just the local filesystem, but also remote services such as Amazon S3. The same goes for writing the executed notebook to a file system: I/O handlers allow papermill to write files to S3 or otherwise.
Writing your own I/O handler requires writing a class that has four methods. All I/O handlers should implement the following class methods:
CustomIO.read(file_path)
, returning the file contentCustomIO.write(file_content, file_path)
, returning nothingCustomIO.pretty_path(path)
, returning a prettified pathCustomIO.listdir(path)
, returning a list of paths.
Note
If you don’t want to support things such as read
because your I/O
handler is only intended for writing (such as a publish-only platform), then
you should implement the method but raise an exception when it is used.
Once you have developed a new handler, you need to declare papermill entry
points in your setup.py
file.
This is done by including the entry_points
key-word argument to setup
in your setup.py file:
from setuptools import setup, find_packages
setup(
# all the normal setup.py arguments...
entry_points={"papermill.io": ["sftp://=papermill_sftp:SFTPHandler"]},
)
This indicates to papermill that when a file path begins with sftp://
, it
should use the class papermill_sftp.SFTPHandler
to handle reading or writing
to that path. Anything before the equal sign is the path prefix, and everything
after it is the class to be used, including where it is imported from.
Traditionally, entry points for papermill I/O handlers look like URL prefixes.
For example, the Amazon Web Services S3 handler is registered under s3://
,
and so is used whenever a path begins with s3://
.
As an example, let’s go through how we would create an I/O handler that reads from an sftp server and writes back to it, so we could do the following:
papermill sftp://my_ftp_server.co.uk/input.ipynb sftp://my_ftp_server.co.uk/output.ipynb
Our project structure will look like this:
papermill_sftp
|- setup.py
|- src
|- papermill_sftp
|- __init__.py
We can define the I/O handler in src/papermill_sftp/__init__.py
. To do so,
we have to create a class that does the relevant actions.
For reading, we will download the file to a temporary path and read it in from there. For writing, we will write to a temporary path and upload it from there. Prettifying the path doesn’t need to change the path, and we are not going to implement a listdir option for now.
import os
import pysftp
sftp_username = os.getenv('SFTP_USERNAME')
sftp_password = os.getenv('SFTP_PASSWORD')
class SFTPHandler:
@classmethod
def read(cls, path):
"""
Read a notebook from an SFTP server.
"""
parsed_url = urllib.parse.urlparse(path)
with tempfile.TemporaryDirectory() as tmpdir:
tmp_file = pathlib.Path(tmpdir) / pathlib.Path(parsed_url.path).name
with pysftp.Connection(
parsed_url.hostname,
username=sftp_username,
password=sftp_password,
port=(parsed_url.port or 22),
cnopts=cnopts,
) as sftp:
sftp.get(parsed_url.path, str(tmp_file))
return tmp_file.read_text()
@classmethod
def write(cls, file_content, path):
"""
Write a notebook to an SFTP server.
"""
parsed_url = urllib.parse.urlparse(path)
with tempfile.TemporaryDirectory() as tmpdir:
tmp_file = pathlib.Path(tmpdir) / "output.ipynb"
tmp_file.write_text(file_content)
with pysftp.Connection(
parsed_url.hostname,
username=sftp_username,
password=sftp_password,
port=(parsed_url.port or 22),
cnopts=cnopts,
) as sftp:
sftp.put(str(tmp_file), parsed_url.path)
@classmethod
def pretty_path(cls, path):
return path
@classmethod
def listdir(cls, path):
raise NotImplementedError
The setup.py
file contains the following code:
from setuptools import setup, find_packages
setup(
name="papermill_sftp",
version="0.1",
url="https://github.com/my_username/papermill_sftp.git",
author="My Name",
author_email="my.email@gmail.com",
description="An SFTP I/O handler for papermill.",
packages=find_packages("./src"),
package_dir={"": "src"},
install_requires=["pysftp"],
entry_points={"papermill.io": ["sftp://=papermill_sftp:SFTPHandler"]},
)
When executing, papermill will check if the input or output path begin with
sftp://
, and if so, use the SFTPHandler from the papermill_sftp project.
A papermill engine is a python object that can run, or execute, a notebook. The default implementation in papermill for example takes in a notebook object, and runs it locally on your machine.
By writing a custom engine, you could allow execution to be handled remotely, or you could apply post-processing to the executed notebook. In the next section, you will see a demonstration.
Papermill engines need to inherit from the papermill.engines.Engine
class.
In order to be used, the new class needs to implement the class method
execute_managed_notebook
. The call signature should match that of the parent
class:
class CustomEngine(papermill.engines.Engine):
@classmethod
execute_managed_notebook(cls, nb_man, kernel_name, **kwargs):
pass
nb_man
is a nbformat.NotebookNode
object, and kernel_name
is a string. Your
custom class then needs to implement the execution of the notebook. For example,
you could insert code that executes the notebook remotely on a server, or
executes the notebook many times to simulate different conditions.
As an example, the following project implements a custom engine that adds the time it took to execute each cell as additional output after every code cell.
The project structure is:
papermill_timing
|- setup.py
|- src
|- papermill_timing
|- __init__.py
The file src/papermill_timing/__init__.py
will implement the engine. Since
papermill already stores information about execution timing in the metadata,
we can leverage the default engine. We will also need to use the nbformat
library to create a notebook node object.
from datetime import datetime
from papermill.engines import NBClientEngine
from nbformat.v4 import new_output
class CustomEngine(NBClientEngine):
@classmethod
def execute_managed_notebook(cls, nb_man, kernel_name, **kwargs):
# call the papermill execution engine:
super().execute_managed_notebook(nb_man, kernel_name, **kwargs)
for cell in nb_man.nb.cells:
if cell.cell_type == "code" and cell.execution_count is not None:
start = datetime.fromisoformat(cell.metadata.papermill.start_time)
end = datetime.fromisoformat(cell.metadata.papermill.end_time)
output_message = f"Execution took {(end - start).total_seconds():.3f} seconds"
output_node = new_output("display_data", data={"text/plain": [output_message]})
cell.outputs = [output_node] + cell.outputs
Once this is in place, we need to add our engine as an entry point to our
setup.py
script - for this, see the following section.
Custom engines can be specified as entry points, under the
papermill.engine
prefix. The entry point needs to reference the class that
we have just implemented. For example, if you write an engine called
TimingEngine in a package called papermill_timing, then in the setup.py
file, you should specify:
from setuptools import setup, find_packages
setup(
name="papermill_timing",
version="0.1",
url="https://github.com/my_username/papermill_timing.git",
author="My Name",
author_email="my.email@gmail.com",
description="A papermill engine that logs additional timing information about code.",
packages=find_packages("./src"),
package_dir={"": "src"},
install_requires=["papermill", "nbformat"],
entry_points={"papermill.engine": ["timer_engine=papermill_timing:TimingEngine"]},
)
This allows users to specify the engine from papermill_timing
by passing the
command line argument --engine timer_engine
.
In the image below, the notebook on the left was executed with the new custom engine, while the one on the left was executed with the standard papermill engine. As you can see, this adds our “injected” output to each code cell
Extending papermill by contributing to it¶
If you find that you’d like to not only add I/O and execution handlers, but think a fundamental aspect of the project could use some improvement, then you may want to contribute to it.
Development of papermill happens on github, and a detailed guide to contributing to it can be found there. There is also a code of conduct there. Please read both documents before beginning!
Troubleshooting¶
NoSuchKernel Errors (using Conda)¶
NoSuchKernel
Errors can appear when running papermill on jupyter notebooks whose
kernel has been specified via conda (nb_conda). nb_conda is used to easily set
conda environment per notebook from within jupyterlab.
To illustrate, the following example demonstrates the creation of a new environment with all the dependencies necessary for an analysis.
conda create -n analysis_1 python=2 ipykernel
Once nb_conda is used within the jupyter server to set the kernel for a notebook to analysis_1, the notebook gets metadata similar to the following:
{
"kernelspec": {
"display_name": "Python [conda env:analysis_1]",
"language": "python",
"name": "conda-env-analysis_1-py"
}
}
Papermill cannot use this metadata to determine that it should use analysis_1 to execute this notebook. Running papermill (from analysis_1 or another environment) will raise the following error:
jupyter_client.kernelspec.NoSuchKernel: No such kernel named conda-env-analysis_1-py
This can be fixed by:
- Installing jupyter (or at least ipykernel) in analysis_1
conda install -n analysis_1 jupyter
- Expose the analysis_1 environment as a jupyter kernel (this is no longer automatic).
conda activate analysis_1
jupyter kernelspec install --user --name analysis_1
- Run papermill (from any environment) specifying the correct kernel using the
-k
option
papermill my_notebook.ipynb output_notebook.ipynb -k analysis_1
API Reference¶
If you are looking for information about a specific function, class, or method, this documentation section will help you.
Reference¶
This part of the documentation lists the full API reference of all public classes and functions.
CLI¶
Command Line options¶
Usage: papermill [OPTIONS] NOTEBOOK_PATH OUTPUT_PATH
This utility executes a single notebook in a subprocess.
Papermill takes a source notebook, applies parameters to the source
notebook, executes the notebook with the specified kernel, and saves the
output in the destination notebook.
The NOTEBOOK_PATH and OUTPUT_PATH can now be replaced by `-` representing
stdout and stderr, or by the presence of pipe inputs / outputs. Meaning
that
`<generate input>... | papermill | ...<process output>`
with `papermill - -` being implied by the pipes will read a notebook from
stdin and write it out to stdout.
Options:
-p, --parameters TEXT... Parameters to pass to the parameters cell.
-r, --parameters_raw TEXT... Parameters to be read as raw string.
-f, --parameters_file TEXT Path to YAML file containing parameters.
-y, --parameters_yaml TEXT YAML string to be used as parameters.
-b, --parameters_base64 TEXT Base64 encoded YAML string as parameters.
--inject-input-path Insert the path of the input notebook as
PAPERMILL_INPUT_PATH as a notebook
parameter.
--inject-output-path Insert the path of the output notebook as
PAPERMILL_OUTPUT_PATH as a notebook
parameter.
--inject-paths Insert the paths of input/output notebooks
as
PAPERMILL_INPUT_PATH/PAPERMILL_OUTPUT_PATH
as notebook parameters.
--engine TEXT The execution engine name to use in
evaluating the notebook.
--request-save-on-cell-execute / --no-request-save-on-cell-execute
Request save notebook after each cell
execution
--prepare-only / --prepare-execute
Flag for outputting the notebook without
execution, but with parameters applied.
-k, --kernel TEXT Name of kernel to run.
--cwd TEXT Working directory to run notebook in.
--progress-bar / --no-progress-bar
Flag for turning on the progress bar.
--log-output / --no-log-output Flag for writing notebook output to the
configured logger.
--stdout-file FILENAME File to write notebook stdout output to.
--stderr-file FILENAME File to write notebook stderr output to.
--log-level [NOTSET|DEBUG|INFO|WARNING|ERROR|CRITICAL]
Set log level
--start_timeout INTEGER Time in seconds to wait for kernel to start.
--execution_timeout INTEGER Time in seconds to wait for each cell before
failing execution (default: forever)
--report-mode / --no-report-mode
Flag for hiding input.
--version Flag for displaying the version.
-h, --help Show this message and exit.
Workflow¶
papermill.engines¶
Engines to perform different roles
-
class
papermill.engines.
Engine
¶ Bases:
object
Base class for engines.
Other specific engine classes should inherit and implement the execute_managed_notebook method.
Defines execute_notebook method which is used to correctly setup the NotebookExecutionManager object for engines to interact against.
-
classmethod
execute_managed_notebook
(nb_man, kernel_name, **kwargs)¶ An abstract method where implementation will be defined in a subclass.
-
classmethod
execute_notebook
(nb, kernel_name, output_path=None, progress_bar=True, log_output=False, autosave_cell_every=30, **kwargs)¶ A wrapper to handle notebook execution tasks.
Wraps the notebook object in a NotebookExecutionManager in order to track execution state in a uniform manner. This is meant to help simplify engine implementations. This allows a developer to just focus on iterating and executing the cell contents.
-
classmethod
-
class
papermill.engines.
NBClientEngine
¶ Bases:
papermill.engines.Engine
A notebook engine representing an nbclient process.
This can execute a notebook document and update the nb_man.nb object with the results.
-
classmethod
execute_managed_notebook
(nb_man, kernel_name, log_output=False, stdout_file=None, stderr_file=None, start_timeout=60, execution_timeout=None, **kwargs)¶ Performs the actual execution of the parameterized notebook locally.
Parameters: - nb (NotebookNode) – Executable notebook object.
- kernel_name (str) – Name of kernel to execute the notebook against.
- log_output (bool) – Flag for whether or not to write notebook output to the configured logger.
- start_timeout (int) – Duration to wait for kernel start-up.
- execution_timeout (int) – Duration to wait before failing execution (default: never).
-
classmethod
-
class
papermill.engines.
NotebookExecutionManager
(nb, output_path=None, log_output=False, progress_bar=True, autosave_cell_every=30)¶ Bases:
object
Wrapper for execution state of a notebook.
This class is a wrapper for notebook objects to house execution state related to the notebook being run through an engine.
In particular the NotebookExecutionManager provides common update callbacks for use within engines to facilitate metadata and persistence actions in a shared manner.
-
COMPLETED
= 'completed'¶
-
FAILED
= 'failed'¶
-
PENDING
= 'pending'¶
-
RUNNING
= 'running'¶
-
autosave_cell
()¶ Saves the notebook if it’s been more than self.autosave_cell_every seconds since it was last saved.
-
cell_complete
(cell, cell_index=None, **kwargs)¶ Finalize metadata for a cell and save notebook.
Optionally called by engines during execution to finalize the metadata for a cell and save the notebook to the output path.
-
cell_exception
(cell, cell_index=None, **kwargs)¶ Set metadata when an exception is raised.
Called by engines when an exception is raised within a notebook to set the metadata on the notebook indicating the location of the failure.
-
cell_start
(cell, cell_index=None, **kwargs)¶ Set and save a cell’s start state.
Optionally called by engines during execution to initialize the metadata for a cell and save the notebook to the output path.
-
cleanup_pbar
()¶ Clean up a progress bar
-
complete_pbar
()¶ Refresh progress bar
-
notebook_complete
(**kwargs)¶ Finalize the metadata for a notebook and save the notebook to the output path.
Called by Engine when execution concludes, regardless of exceptions.
-
notebook_start
(**kwargs)¶ Initialize a notebook, clearing its metadata, and save it.
When starting a notebook, this initializes and clears the metadata for the notebook and its cells, and saves the notebook to the given output path.
Called by Engine when execution begins.
-
now
()¶ Helper to return current UTC time
-
save
(**kwargs)¶ Saves the wrapped notebook state.
If an output path is known, this triggers a save of the wrapped notebook state to the provided path.
Can be used outside of cell state changes if execution is taking a long time to conclude but the notebook object should be synced.
For example, you may want to save the notebook every 10 minutes when running a 5 hour cell execution to capture output messages in the notebook.
-
set_timer
()¶ Initializes the execution timer for the notebook.
This is called automatically when a NotebookExecutionManager is constructed.
-
-
class
papermill.engines.
PapermillEngines
¶ Bases:
object
The holder which houses any engine registered with the system.
This object is used in a singleton manner to save and load particular named Engine objects so they may be referenced externally.
-
execute_notebook_with_engine
(engine_name, nb, kernel_name, **kwargs)¶ Fetch a named engine and execute the nb object against it.
-
get_engine
(name=None)¶ Retrieves an engine by name.
-
register
(name, engine)¶ Register a named engine
-
register_entry_points
()¶ Register entrypoints for an engine
Load handlers provided by other packages
-
-
papermill.engines.
catch_nb_assignment
(func)¶ Wrapper to catch nb keyword arguments
This helps catch nb keyword arguments and assign onto self when passed to the wrapped function.
Used for callback methods when the caller may optionally have a new copy of the originally wrapped nb object.
papermill.execute¶
-
papermill.execute.
execute_notebook
(input_path, output_path, parameters=None, engine_name=None, request_save_on_cell_execute=True, prepare_only=False, kernel_name=None, progress_bar=True, log_output=False, stdout_file=None, stderr_file=None, start_timeout=60, report_mode=False, cwd=None, **engine_kwargs)¶ Executes a single notebook locally.
Parameters: - input_path (str) – Path to input notebook
- output_path (str) – Path to save executed notebook
- parameters (dict, optional) – Arbitrary keyword arguments to pass to the notebook parameters
- engine_name (str, optional) – Name of execution engine to use
- request_save_on_cell_execute (bool, optional) – Request save notebook after each cell execution
- autosave_cell_every (int, optional) – How often in seconds to save in the middle of long cell executions
- prepare_only (bool, optional) – Flag to determine if execution should occur or not
- kernel_name (str, optional) – Name of kernel to execute the notebook against
- progress_bar (bool, optional) – Flag for whether or not to show the progress bar.
- log_output (bool, optional) – Flag for whether or not to write notebook output to the configured logger
- start_timeout (int, optional) – Duration in seconds to wait for kernel start-up
- report_mode (bool, optional) – Flag for whether or not to hide input.
- cwd (str, optional) – Working directory to use when executing the notebook
- **kwargs – Arbitrary keyword arguments to pass to the notebook engine
Returns: nb – Executed notebook object
Return type: NotebookNode
-
papermill.execute.
prepare_notebook_metadata
(nb, input_path, output_path, report_mode=False)¶ Prepare metadata associated with a notebook and its cells
Parameters:
-
papermill.execute.
raise_for_execution_errors
(nb, output_path)¶ Assigned parameters into the appropriate place in the input notebook
Parameters: - nb (NotebookNode) – Executable notebook object
- output_path (str) – Path to write executed notebook
-
papermill.execute.
remove_error_markers
(nb)¶
papermill.clientwrap¶
-
class
papermill.clientwrap.
PapermillNotebookClient
(nb_man, km=None, raise_on_iopub_timeout=True, **kw)¶ Bases:
nbclient.client.NotebookClient
Module containing a that executes the code cells and updates outputs
-
execute
(**kwargs)¶ Wraps the parent class process call slightly
-
log_output
¶ A boolean (True, False) trait.
-
log_output_message
(output)¶ Process a given output. May log it in the configured logger and/or write it into the configured stdout/stderr files.
Parameters: output – nbformat.notebooknode.NotebookNode Returns:
-
papermill_execute_cells
()¶ This function replaces cell execution with it’s own wrapper.
We are doing this for the following reasons:
- Notebooks will stop executing when they encounter a failure but not raise a CellException. This allows us to save the notebook with the traceback even though a CellExecutionError was encountered.
- We want to write the notebook as cells are executed. We inject our logic for that here.
- We want to include timing and execution status information with the metadata of each cell.
-
process_message
(*arg, **kwargs)¶ Processes a kernel message, updates cell state, and returns the resulting output object that was appended to cell.outputs.
The input argument cell is modified in-place.
Parameters: Returns: output – The execution output payload (or None for no output).
Return type: Raises: CellExecutionComplete
– Once a message arrives which indicates computation completeness.
-
stderr_file
¶ A trait whose value must be an instance of a specified class.
The value can also be an instance of a subclass of the specified class.
Subclasses can declare default classes by overriding the klass attribute
-
stdout_file
¶ A trait whose value must be an instance of a specified class.
The value can also be an instance of a subclass of the specified class.
Subclasses can declare default classes by overriding the klass attribute
-
Language Translators¶
Translators¶
Translator¶
-
class
papermill.translators.
Translator
¶ -
classmethod
assign
(name, str_val)¶
-
classmethod
codify
(parameters, comment='Parameters')¶
-
classmethod
comment
(cmt_str)¶
-
classmethod
translate
(val)¶ Translate each of the standard json/yaml types to appropiate objects.
-
classmethod
translate_bool
(val)¶ Default behavior for translation
-
classmethod
translate_dict
(val)¶
-
classmethod
translate_escaped_str
(str_val)¶ Reusable by most interpreters
-
classmethod
translate_float
(val)¶ Default behavior for translation
-
classmethod
translate_int
(val)¶ Default behavior for translation
-
classmethod
translate_list
(val)¶
-
classmethod
translate_none
(val)¶ Default behavior for translation
-
classmethod
translate_raw_str
(val)¶ Reusable by most interpreters
-
classmethod
translate_str
(val)¶ Default behavior for translation
-
classmethod
PapermillTranslators¶
-
class
papermill.translators.
PapermillTranslators
¶ The holder which houses any translator registered with the system. This object is used in a singleton manner to save and load particular named Translator objects for reference externally.
-
find_translator
(kernel_name, language)¶
-
register
(language, translator)¶
-
Python¶
-
class
papermill.translators.
PythonTranslator
¶ -
classmethod
codify
(parameters, comment='Parameters')¶
-
classmethod
comment
(cmt_str)¶
-
classmethod
translate_bool
(val)¶ Default behavior for translation
-
classmethod
translate_dict
(val)¶
-
classmethod
translate_float
(val)¶ Default behavior for translation
-
classmethod
translate_list
(val)¶
-
classmethod
R¶
-
class
papermill.translators.
RTranslator
¶ -
classmethod
assign
(name, str_val)¶
-
classmethod
comment
(cmt_str)¶
-
classmethod
translate_bool
(val)¶ Default behavior for translation
-
classmethod
translate_dict
(val)¶
-
classmethod
translate_list
(val)¶
-
classmethod
translate_none
(val)¶ Default behavior for translation
-
classmethod
Julia¶
Scala¶
Input / Output¶
papermill.iorw¶
-
class
papermill.iorw.
ABSHandler
¶ Bases:
object
-
listdir
(path)¶
-
pretty_path
(path)¶
-
read
(path)¶
-
write
(buf, path)¶
-
-
class
papermill.iorw.
ADLHandler
¶ Bases:
object
-
listdir
(path)¶
-
pretty_path
(path)¶
-
read
(path)¶
-
write
(buf, path)¶
-
-
class
papermill.iorw.
GCSHandler
¶ Bases:
object
-
RATE_LIMIT_RETRIES
= 3¶
-
RETRY_DELAY
= 1¶
-
RETRY_MAX_DELAY
= 4¶
-
RETRY_MULTIPLIER
= 1¶
-
listdir
(path)¶
-
pretty_path
(path)¶
-
read
(path)¶
-
write
(buf, path)¶
-
-
class
papermill.iorw.
HDFSHandler
¶ Bases:
object
-
listdir
(path)¶
-
pretty_path
(path)¶
-
read
(path)¶
-
write
(buf, path)¶
-
-
class
papermill.iorw.
HttpHandler
¶ Bases:
object
-
classmethod
listdir
(path)¶
-
classmethod
pretty_path
(path)¶
-
classmethod
read
(path)¶
-
classmethod
write
(buf, path)¶
-
classmethod
-
class
papermill.iorw.
LocalHandler
¶ Bases:
object
-
cwd
(new_path)¶ Sets the cwd during reads and writes
-
listdir
(path)¶
-
pretty_path
(path)¶
-
read
(path)¶
-
write
(buf, path)¶
-
-
class
papermill.iorw.
NoDatesSafeLoader
(stream)¶ Bases:
yaml.loader.SafeLoader
-
yaml_implicit_resolvers
= {'': [('tag:yaml.org,2002:null', re.compile('^(?: ~\n |null|Null|NULL\n | )$', re.VERBOSE))], '!': [('tag:yaml.org,2002:yaml', re.compile('^(?:!|&|\\*)$'))], '&': [('tag:yaml.org,2002:yaml', re.compile('^(?:!|&|\\*)$'))], '*': [('tag:yaml.org,2002:yaml', re.compile('^(?:!|&|\\*)$'))], '+': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '-': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '.': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE))], '0': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '1': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '2': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '3': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '4': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '5': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '6': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '7': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '8': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '9': [('tag:yaml.org,2002:float', re.compile('^(?:[-+]?(?:[0-9][0-9_]*)\\.[0-9_]*(?:[eE][-+][0-9]+)?\n |\\.[0-9_]+(?:[eE][-+][0-9]+)?\n |[-+]?[0-9][0-9_]*(?::[0-5]?[0-9])+\\.[0-9_]*\n |[-+, re.VERBOSE)), ('tag:yaml.org,2002:int', re.compile('^(?:[-+]?0b[0-1_]+\n |[-+]?0[0-7_]+\n |[-+]?(?:0|[1-9][0-9_]*)\n |[-+]?0x[0-9a-fA-F_]+\n |[-+]?[1-9][0-9_]*(?::[0-5]?[0-9]), re.VERBOSE))], '<': [('tag:yaml.org,2002:merge', re.compile('^(?:<<)$'))], '=': [('tag:yaml.org,2002:value', re.compile('^(?:=)$'))], 'F': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], 'N': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE)), ('tag:yaml.org,2002:null', re.compile('^(?: ~\n |null|Null|NULL\n | )$', re.VERBOSE))], 'O': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], 'T': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], 'Y': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], 'f': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], 'n': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE)), ('tag:yaml.org,2002:null', re.compile('^(?: ~\n |null|Null|NULL\n | )$', re.VERBOSE))], 'o': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], 't': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], 'y': [('tag:yaml.org,2002:bool', re.compile('^(?:yes|Yes|YES|no|No|NO\n |true|True|TRUE|false|False|FALSE\n |on|On|ON|off|Off|OFF)$', re.VERBOSE))], '~': [('tag:yaml.org,2002:null', re.compile('^(?: ~\n |null|Null|NULL\n | )$', re.VERBOSE))]}¶
-
-
class
papermill.iorw.
PapermillIO
¶ Bases:
object
The holder which houses any io system registered with the system. This object is used in a singleton manner to save and load particular named Handler objects for reference externally.
-
get_handler
(path)¶
-
listdir
(path)¶
-
pretty_path
(path)¶
-
read
(path, extensions=['.ipynb', '.json'])¶
-
register
(scheme, handler)¶
-
register_entry_points
()¶
-
reset
()¶
-
write
(buf, path, extensions=['.ipynb', '.json'])¶
-
-
class
papermill.iorw.
S3Handler
¶ Bases:
object
-
classmethod
listdir
(path)¶
-
classmethod
pretty_path
(path)¶
-
classmethod
read
(path)¶
-
classmethod
write
(buf, path)¶
-
classmethod
-
papermill.iorw.
fallback_gs_is_retriable
(e)¶
-
papermill.iorw.
get_pretty_path
(path)¶
-
papermill.iorw.
gs_is_retriable
(e)¶
-
papermill.iorw.
list_notebook_files
(path)¶ Returns a list of all the notebook files in a directory.
-
papermill.iorw.
load_notebook_node
(notebook_path)¶ Returns a notebook object with papermill metadata loaded from the specified path.
Parameters: notebook_path (str) – Path to the notebook file. Returns: nbformat.NotebookNode
-
papermill.iorw.
local_file_io_cwd
(path=None)¶
-
papermill.iorw.
read_yaml_file
(path)¶ Reads a YAML file from the location specified at ‘path’.
-
papermill.iorw.
write_ipynb
(nb, path)¶ Saves a notebook object to the specified path. :param nb_node: Notebook object to save. :type nb_node: nbformat.NotebookNode :param notebook_path: Path to save the notebook object to. :type notebook_path: str
Storage¶
Utilities¶
Utils¶
-
papermill.utils.
chdir
(path)¶ Change working directory to path and restore old path on exit.
path can be None in which case this is a no-op.
-
papermill.utils.
merge_kwargs
(caller_args, **callee_args)¶ Merge named argument.
Function takes a dictionary of caller arguments and callee arguments as keyword arguments Returns a dictionary with merged arguments. If same argument is in both caller and callee arguments the last one will be taken and warning will be raised.
Parameters: - caller_args (dict) – Caller arguments
- **callee_args – Keyword callee arguments
Returns: args – Merged arguments
Return type:
-
papermill.utils.
remove_args
(args=None, **kwargs)¶ Remove arguments from kwargs.
Parameters: - args (list) – Argument names to remove from kwargs
- **kwargs – Arbitrary keyword arguments
Returns: kwargs – New dictionary of arguments
Return type:
-
papermill.utils.
retry
(num)¶
Exceptions¶
-
exception
papermill.exceptions.
AwsError
¶ Raised when an AWS Exception is encountered.
-
exception
papermill.exceptions.
FileExistsError
¶ Raised when a File already exists on S3.
-
exception
papermill.exceptions.
PapermillException
¶ Raised when an exception is encountered when operating on a notebook.
-
exception
papermill.exceptions.
PapermillExecutionError
(cell_index, exec_count, source, ename, evalue, traceback)¶ Raised when an exception is encountered in a notebook.
-
exception
papermill.exceptions.
PapermillMissingParameterException
¶ Raised when a parameter without a value is required to operate on a notebook.
-
exception
papermill.exceptions.
PapermillOptionalDependencyException
¶ Raised when an exception is encountered when an optional plugin is missing.
-
exception
papermill.exceptions.
PapermillParameterOverwriteWarning
¶ Callee overwrites caller argument to pass down the stream.
-
exception
papermill.exceptions.
PapermillRateLimitException
¶ Raised when an io request has been rate limited
-
exception
papermill.exceptions.
PapermillWarning
¶ Base warning for papermill.
-
papermill.exceptions.
missing_dependency_generator
(package, dep)¶
-
papermill.exceptions.
missing_environment_variable_generator
(package, env_key)¶
Log¶
Sets up a logger
papermill.tests package¶
Submodules¶
papermill.tests.test_abs module¶
papermill.tests.test_adl module¶
papermill.tests.test_autosave module¶
papermill.tests.test_cli module¶
papermill.tests.test_clientwrap module¶
papermill.tests.test_conf module¶
papermill.tests.test_engines module¶
-
papermill.tests.test_engines.
AnyMock
(cls)¶ Mocks a matcher for any instance of class cls. e.g. my_mock.called_once_with(Any(int), “bar”)
-
class
papermill.tests.test_engines.
TestEngineBase
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
test_cell_callback_execute
()¶
-
test_no_cell_callback_execute
()¶
-
test_wrap_and_execute_notebook
()¶ Mocks each wrapped call and proves the correct inputs get applied to the correct underlying calls for execute_notebook.
-
-
class
papermill.tests.test_engines.
TestEngineRegistration
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
test_getting
()¶
-
test_registering_entry_points
()¶
-
test_registration
()¶
-
-
class
papermill.tests.test_engines.
TestNBClientEngine
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
test_nb_convert_engine
()¶
-
test_nb_convert_engine_execute
()¶
-
test_nb_convert_log_outputs
()¶
-
test_nb_convert_no_log_outputs
()¶
-
-
class
papermill.tests.test_engines.
TestNotebookExecutionManager
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
test_basic_pbar
()¶
-
test_cell_complete_after_cell_exception
()¶
-
test_cell_complete_after_cell_start
()¶
-
test_cell_complete_new_nb
()¶
-
test_cell_complete_without_cell_start
()¶
-
test_cell_exception
()¶
-
test_cell_exception_new_nb
()¶
-
test_cell_start
()¶
-
test_cell_start_new_nb
()¶
-
test_nb_isolation
()¶ Tests that the engine notebook is isolated from source notebook
-
test_no_pbar
()¶
-
test_notebook_complete
()¶
-
test_notebook_complete_cell_status_completed
()¶
-
test_notebook_complete_cell_status_with_failed
()¶
-
test_notebook_complete_new_nb
()¶
-
test_notebook_start
()¶
-
test_notebook_start_markdown_code
()¶
-
test_notebook_start_new_nb
()¶
-
test_save
()¶
-
test_save_new_nb
()¶
-
test_save_no_output
()¶
-
test_set_timer
()¶
-
papermill.tests.test_exceptions module¶
papermill.tests.test_execute module¶
-
class
papermill.tests.test_execute.
TestBrokenNotebook1
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
tearDown
()¶ Hook method for deconstructing the test fixture after testing it.
-
test
()¶
-
-
class
papermill.tests.test_execute.
TestBrokenNotebook2
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
tearDown
()¶ Hook method for deconstructing the test fixture after testing it.
-
test
()¶
-
-
class
papermill.tests.test_execute.
TestCWD
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
tearDown
()¶ Hook method for deconstructing the test fixture after testing it.
-
test_execution_respects_cwd_assignment
()¶
-
test_local_save_ignores_cwd_assignment
()¶
-
-
class
papermill.tests.test_execute.
TestNotebookHelpers
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
tearDown
()¶ Hook method for deconstructing the test fixture after testing it.
-
test_backslash_params
()¶
-
test_backslash_quote_params
()¶
-
test_cell_insertion
()¶
-
test_default_start_timeout
(preproc_mock)¶
-
test_double_backslash_quote_params
()¶
-
test_prepare_only
()¶
-
test_quoted_params
()¶
-
test_start_timeout
(preproc_mock)¶
-
-
class
papermill.tests.test_execute.
TestReportMode
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
tearDown
()¶ Hook method for deconstructing the test fixture after testing it.
-
test_report_mode
()¶
-
-
class
papermill.tests.test_execute.
TestSysExit
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
tearDown
()¶ Hook method for deconstructing the test fixture after testing it.
-
test_sys_exit
()¶
-
test_sys_exit0
()¶
-
test_sys_exit1
()¶
-
-
papermill.tests.test_execute.
execute_notebook
(input_path, output_path, parameters=None, engine_name=None, request_save_on_cell_execute=True, prepare_only=False, *, kernel_name='python3', progress_bar=True, log_output=False, stdout_file=None, stderr_file=None, start_timeout=60, report_mode=False, cwd=None, **engine_kwargs)¶ Executes a single notebook locally.
Parameters: - input_path (str) – Path to input notebook
- output_path (str) – Path to save executed notebook
- parameters (dict, optional) – Arbitrary keyword arguments to pass to the notebook parameters
- engine_name (str, optional) – Name of execution engine to use
- request_save_on_cell_execute (bool, optional) – Request save notebook after each cell execution
- autosave_cell_every (int, optional) – How often in seconds to save in the middle of long cell executions
- prepare_only (bool, optional) – Flag to determine if execution should occur or not
- kernel_name (str, optional) – Name of kernel to execute the notebook against
- progress_bar (bool, optional) – Flag for whether or not to show the progress bar.
- log_output (bool, optional) – Flag for whether or not to write notebook output to the configured logger
- start_timeout (int, optional) – Duration in seconds to wait for kernel start-up
- report_mode (bool, optional) – Flag for whether or not to hide input.
- cwd (str, optional) – Working directory to use when executing the notebook
- **kwargs – Arbitrary keyword arguments to pass to the notebook engine
Returns: nb – Executed notebook object
Return type: NotebookNode
papermill.tests.test_gcs module¶
-
class
papermill.tests.test_gcs.
GCSTest
(methodName='runTest')¶ Bases:
unittest.case.TestCase
Tests for GCS.
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
test_fallback_gcs_invalid_code
(mock_gcs_filesystem, mock_gcs_retriable)¶
-
test_gcs_fallback_retry_unknown_failure_code
(mock_gcs_filesystem, mock_gcs_retriable)¶
-
test_gcs_handle_exception
(mock_gcs_filesystem)¶
-
test_gcs_invalid_code
(mock_gcs_filesystem, mock_gcs_retriable)¶
-
test_gcs_listdir
(mock_gcs_filesystem)¶
-
test_gcs_read
(mock_gcs_filesystem)¶
-
test_gcs_retry
(mock_gcs_filesystem)¶
-
test_gcs_retry_older_exception
(mock_gcs_filesystem)¶
-
test_gcs_unretryable
(mock_gcs_filesystem)¶
-
test_gcs_write
(mock_gcs_filesystem)¶
-
-
class
papermill.tests.test_gcs.
MockGCSFile
(exception=None, max_raises=1)¶ Bases:
object
-
read
()¶
-
write
(buf)¶
-
-
papermill.tests.test_gcs.
mock_gcs_fs_wrapper
(exception=None, max_raises=1)¶
papermill.tests.test_hdfs module¶
-
class
papermill.tests.test_hdfs.
HDFSTest
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
setUp
()¶ Hook method for setting up the test fixture before exercising it.
-
test_hdfs_listdir
(mock_hdfs_filesystem)¶
-
test_hdfs_read
(mock_hdfs_filesystem)¶
-
test_hdfs_write
(mock_hdfs_filesystem)¶
-
-
class
papermill.tests.test_hdfs.
MockHadoopFileSystem
(*args, **kw)¶ Bases:
unittest.mock.MagicMock
-
ls
(path)¶
-
open
(path, *args)¶
-
papermill.tests.test_iorw module¶
papermill.tests.test_parameterize module¶
-
class
papermill.tests.test_parameterize.
TestBuiltinParameters
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
test_add_builtin_parameters_adds_dict_of_builtins
()¶
-
test_add_builtin_parameters_allows_to_override_builtin
()¶
-
test_add_builtin_parameters_keeps_provided_parameters
()¶
-
test_builtin_parameters_include_current_datetime_local
()¶
-
test_builtin_parameters_include_current_datetime_utc
()¶
-
test_builtin_parameters_include_run_uuid
()¶
-
-
class
papermill.tests.test_parameterize.
TestNotebookParametrizing
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
count_nb_injected_parameter_cells
(nb)¶
-
test_custom_comment
()¶
-
test_injected_parameters_tag
()¶
-
test_no_parameter_tag
()¶
-
test_no_tag_copying
()¶
-
test_repeated_run_injected_parameters_tag
()¶
-
test_repeated_run_no_parameters_tag
()¶
-
-
class
papermill.tests.test_parameterize.
TestPathParameterizing
(methodName='runTest')¶ Bases:
unittest.case.TestCase
-
test_parameterized_path_with_none_parameters
()¶
-
test_parameterized_path_with_undefined_parameter
()¶
-
test_path_with_boolean_parameter
()¶
-
test_path_with_dict_parameter
()¶
-
test_path_with_float_format_string
()¶
-
test_path_with_list_parameter
()¶
-
test_path_with_multiple_parameter
()¶
-
test_path_with_none_parameter
()¶
-
test_path_with_numeric_format_string
()¶
-
test_path_with_numeric_parameter
()¶
-
test_path_with_single_parameter
()¶
-
test_plain_text_path_with_empty_parameters_object
()¶
-
test_plain_text_path_with_none_parameters
()¶
-
test_plain_text_path_with_unused_parameters
()¶
-