Welcome to papermill

Papermill is a tool for parameterizing and executing Jupyter Notebooks.

Papermill lets you:

• parameterize notebooks
• execute notebooks

This opens up new opportunities for how notebooks can be used. For example:

• Perhaps you have a financial report that you wish to run with different values on the first or last day of a month or at the beginning or end of the year, using parameters makes this task easier.
• Do you want to run a notebook and depending on its results, choose a particular notebook to run next? You can now programmatically execute a workflow without having to copy and paste from notebook to notebook manually.

Python Version Support

This library will support python 2.7 and 3.5+ until end-of-life for python 2 in 2020. After which python 2 support will halt and only 3.x version will be maintained.

Documentation

These pages guide you through the installation and usage of papermill.

Installation

Installing papermill

From the command line:

Python 3

python3 -m pip install papermill

Python 2

python -m pip install papermill

If unsure whether to use Python 3 or Python 2, we recommend using Python 3.

Installing In-Notebook language bindings

In-Notebook language bindings provide helpers and utilities for using Papermill with a programming language.

Python bindings

No additional installation steps are required since Python bindings are built into papermill.

R bindings

The R language bindings are provided by the papermillr project. Follow installation instructions for R language bindings.

Usage

For an interactive example that demonstrates the usage of papermill, click the Binder link below:

Using papermill

The general workflow when using papermill is parameterizing a notebook, executing it, as well as storing the results. In addition to operating on a single notebook, papermill also works on a collection of notebooks.

Parameterize

See also

Workflow reference

Generally, the first workflow step when using papermill is to parameterize the notebook.

To do this, you will tag notebook cells with parameters. These parameters are later used when the notebook is executed or run.

Designate parameters for a cell

To parameterize your notebook, designate a cell with the tag parameters.

How do parameters work

Papermill looks for the parameters cell and treats those values as defaults for the parameters passed in at execution time. It achieves this by inserting a cell after the tagged cell. If no cell is tagged with parameters a cell will be inserted to the front of the notebook.

Execute

The two ways to execute the notebook with parameters are: (1) through the Python API and (2) through the command line interface.

Execute via the Python API

The execute_notebook function can be called to execute an input notebook when passed a dictionary of parameters:

execute_notebook(<input notebook>, <output notebook>, <dictionary of parameters>)

import papermill as pm

pm.execute_notebook(
   'path/to/input.ipynb',
   'path/to/output.ipynb',
   parameters=dict(alpha=0.6, ratio=0.1)
)

Execute via CLI

To execute a notebook using the CLI, enter the papermill command in the terminal with the input notebook, location for output notebook, and options.

See also

CLI reference

Execute a notebook with parameters

Here’s an example of a local notebook being executed and output to an Amazon S3 account:

$ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1

In the above example, two parameters are set: alpha and l1_ratio using -p (--parameters also works). Parameter values that look like booleans or numbers will be interpreted as such.

Here are the different ways users may set parameters:

Using raw strings as parameters

Using -r or --parameters_raw, users can set parameters one by one. However, unlike -p, the parameter will remain a string, even if it may be interpreted as a number or boolean.

$ papermill local/input.ipynb s3://bkt/output.ipynb -r version 1.0

Using a parameters file

Using -f or --parameters_file, users can provide a YAML file from which parameter values should be read.

$ papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml

Using a YAML string for parameters

Using -y or --parameters_yaml, users can directly provide a YAML string containing parameter values.

$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
    - 0.0
    - 1.0
    - 2.0
    - 3.0
linear_function:
    slope: 3.0
    intercept: 1.0"

Using -b or --parameters_base64, users can provide a YAML string, base64-encoded, containing parameter values.

$ papermill local/input.ipynb s3://bkt/output.ipynb -b YWxwaGE6IDAuNgpsMV9yYXRpbzogMC4xCg==

Note about using YAML

When using YAML to pass arguments, through -y, -b or -f, parameter values can be arrays or dictionaries:

$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
    - 0.0
    - 1.0
    - 2.0
    - 3.0
linear_function:
    slope: 3.0
    intercept: 1.0"

Store

See also

Reference - Storage

Papermill can store notebooks in a number of locations including AWS S3, Azure data blobs, and Azure data lakes.

The modular architecture of papermill allows new data stores to be added over time.

Command Line Interface

papermill may be executed from the terminal. The following are the command options:

Usage: papermill [OPTIONS] NOTEBOOK_PATH OUTPUT_PATH

  This utility executes a single notebook on a container.

  Papermill takes a source notebook, applies parameters to the source
  notebook, executes the notebook with the specified kernel, and saves the
  output in the destination notebook.

Options:
  -p, --parameters TEXT...        Parameters to pass to the parameters cell.
  -r, --parameters_raw TEXT...    Parameters to be read as raw string.
  -f, --parameters_file TEXT      Path to YAML file containing parameters.
  -y, --parameters_yaml TEXT      YAML string to be used as parameters.
  -b, --parameters_base64 TEXT    Base64 encoded YAML string as parameters.
  --inject-input-path             Insert the path of the input notebook as
                                  PAPERMILL_INPUT_PATH as a notebook
                                  parameter.
  --inject-output-path            Insert the path of the output notebook as
                                  PAPERMILL_OUTPUT_PATH as a notebook
                                  parameter.
  --inject-paths                  Insert the paths of input/output notebooks
                                  as
                                  PAPERMILL_INPUT_PATH/PAPERMILL_OUTPUT_PATH
                                  as notebook parameters.
  --engine TEXT                   The execution engine name to use in
                                  evaluating the notebook.
  --prepare-only / --prepare-execute
                                  Flag for outputting the notebook without
                                  execution, but with parameters applied.
  -k, --kernel TEXT               Name of kernel to run.
  --cwd TEXT                      Working directory to run notebook in.
  --progress-bar / --no-progress-bar
                                  Flag for turning on the progress bar.
  --log-output / --no-log-output  Flag for writing notebook output to stderr.
  --log-level [NOTSET|DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Set log level
  --start_timeout INTEGER         Time in seconds to wait for kernel to start.
  --report-mode / --not-report-mode
                                  Flag for hiding input.
  --version                       Flag for displaying the version.
  -h, --help                      Show this message and exit.

Extending papermill

Papermill provides some interfaces with external services out of the box. However, you may find that you would like papermill to do more than it currently does. You could contribute to the papermill project yourself (see Extending papermill by contributing to it). However, an easier method might be to extend papermill using entry points.

In general, when you run a notebook with papermill, the following happens:

1. The notebook file is read in
2. The file content is converted to a notebook python object
3. The notebook is executed
4. The notebook is written to a file

Through entry points, you can write your own tools to handle steps 1, 3, and 4. If you find that there’s more you want to contribute to papermill, consider developing papermill itself.

Extending papermill through entry points

What are entry points?

The python packaging documentation describes entry points as:

Entry points are a mechanism for an installed distribution to advertise components it provides to be discovered and used by other code. For example:

Distributions can specify console_scripts entry points, each referring to a function. When pip (or another console_scripts aware installer) installs the distribution, it will create a command-line wrapper for each entry point.

Applications can use entry points to load plugins; e.g. Pygments (a syntax highlighting tool) can use additional lexers and styles from separately installed packages. For more about this, see Creating and discovering plugins.

When running, papermill looks for entry points that implement input / output (I/O) handlers, and execution handlers.

Developing new I/O handlers

Virtually the first thing that happens when papermill is used is that the input notebook is read in. This is managed by I/O handlers, which allow papermill to access not just the local filesystem, but also remote services such as Amazon S3. The same goes for writing the executed notebook to a file system: I/O handlers allow papermill to write files to S3 or otherwise.

Creating a new handler

Writing your own I/O handler requires writing a class that has four methods. All I/O handlers should implement the following class methods:

• CustomIO.read(file_path), returning the file content
• CustomIO.write(file_content, file_path), returning nothing
• CustomIO.pretty_path(path), returning a prettified path
• CustomIO.listdir(path), returning a list of paths.

Note

If you don’t want to support things such as read because your I/O handler is only intended for writing (such as a publish-only platform), then you should implement the method but raise an exception when it is used.

Ensuring your handler is found by papermill

Once you have developed a new handler, you need to declare papermill entry points in your setup.py file.

This is done by including the entry_points key-word argument to setup in your setup.py file:

from setuptools import setup, find_packages
setup(
    # all the normal setup.py arguments...
    entry_points={"papermill.io": ["sftp://=papermill_sftp:SFTPHandler"]},
)

This indicates to papermill that when a file path begins with sftp://, it should use the class papermill_sftp.SFTPHandler to handle reading or writing to that path. Anything before the equal sign is the path prefix, and everything after it is the class to be used, including where it is imported from.

Traditionally, entry points for papermill I/O handlers look like URL prefixes. For example, the Amazon Web Services S3 handler is registered under s3://, and so is used whenever a path begins with s3://.

Example: sftp I/O handler

As an example, let’s go through how we would create an I/O handler that reads from an sftp server and writes back to it, so we could do the following:

papermill sftp://my_ftp_server.co.uk/input.ipynb sftp://my_ftp_server.co.uk/output.ipynb

Our project structure will look like this:

papermill_sftp
    |- setup.py
    |- src
        |- papermill_sftp
            |- __init__.py

We can define the I/O handler in src/papermill_sftp/__init__.py. To do so, we have to create a class that does the relevant actions.

For reading, we will download the file to a temporary path and read it in from there. For writing, we will write to a temporary path and upload it from there. Prettifying the path doesn’t need to change the path, and we are not going to implement a listdir option for now.

import os
import pysftp

sftp_username = os.getenv('SFTP_USERNAME')
sftp_password = os.getenv('SFTP_PASSWORD')

class SFTPHandler:

    @classmethod
    def read(cls, path):
        """
        Read a notebook from an SFTP server.
        """
        parsed_url = urllib.parse.urlparse(path)
        with tempfile.TemporaryDirectory() as tmpdir:
            tmp_file = pathlib.Path(tmpdir) / pathlib.Path(parsed_url.path).name
            with pysftp.Connection(
                parsed_url.hostname,
                username=sftp_username,
                password=sftp_password,
                port=(parsed_url.port or 22),
                cnopts=cnopts,
            ) as sftp:
                sftp.get(parsed_url.path, str(tmp_file))
            return tmp_file.read_text()

    @classmethod
    def write(cls, file_content, path):
        """
        Write a notebook to an SFTP server.
        """
        parsed_url = urllib.parse.urlparse(path)
        with tempfile.TemporaryDirectory() as tmpdir:
            tmp_file = pathlib.Path(tmpdir) / "output.ipynb"
            tmp_file.write_text(file_content)
            with pysftp.Connection(
                parsed_url.hostname,
                username=sftp_username,
                password=sftp_password,
                port=(parsed_url.port or 22),
                cnopts=cnopts,
            ) as sftp:
                sftp.put(str(tmp_file), parsed_url.path)

    @classmethod
    def pretty_path(cls, path):
        return path

    @classmethod
    def listdir(cls, path):
        raise NotImplementedError

The setup.py file contains the following code:

from setuptools import setup, find_packages

setup(
    name="papermill_sftp",
    version="0.1",
    url="https://github.com/my_username/papermill_sftp.git",
    author="My Name",
    author_email="my.email@gmail.com",
    description="An SFTP I/O handler for papermill.",
    packages=find_packages("./src"),
    package_dir={"": "src"},
    install_requires=["pysftp"],
    entry_points={"papermill.io": ["sftp://=papermill_sftp:SFTPHandler"]},
)

When executing, papermill will check if the input or output path begin with sftp://, and if so, use the SFTPHandler from the papermill_sftp project.

Developing a new engine

A papermill engine is a python object that can run, or execute, a notebook. The default implementation in papermill for example takes in a notebook object, and runs it locally on your machine.

By writing a custom engine, you could allow execution to be handled remotely, or you could apply post-processing to the executed notebook. In the next section, you will see a demonstration.

Creating a new engine

Papermill engines need to inherit from the papermill.engines.Engine class.

In order to be used, the new class needs to implement the class method execute_managed_notebook. The call signature should match that of the parent class:

class CustomEngine(papermill.engines.Engine):

    @classmethod
    execute_managed_notebook(cls, nb_man, kernel_name, **kwargs):
        pass

nb_man is a nbformat.NotebookNode object, and kernel_name is a string. Your custom class then needs to implement the execution of the notebook. For example, you could insert code that executes the notebook remotely on a server, or executes the notebook many times to simulate different conditions.

As an example, the following project implements a custom engine that adds the time it took to execute each cell as additional output after every code cell.

The project structure is:

papermill_timing
    |- setup.py
    |- src
        |- papermill_timing
            |- __init__.py

The file src/papermill_timing/__init__.py will implement the engine. Since papermill already stores information about execution timing in the metadata, we can leverage the default engine. We will also need to use the nbformat library to create a notebook node object.

from datetime import datetime
from papermill.engines import NBConvertEngine
from nbformat.v4 import new_output

class CustomEngine(NBConvertEngine):

    @classmethod
    def execute_managed_notebook(cls, nb_man, kernel_name, **kwargs):

        # call the papermill execution engine:
        super().execute_managed_notebook(nb_man, kernel_name, **kwargs)

        for cell in nb_man.nb.cells:

            if cell.cell_type == "code" and cell.execution_count is not None:
                start = datetime.fromisoformat(cell.metadata.papermill.start_time)
                end = datetime.fromisoformat(cell.metadata.papermill.end_time)
                output_message = f"Execution took {(end - start).total_seconds():.3f} seconds"
                output_node = new_output("display_data", data={"text/plain": [output_message]})
                cell.outputs = [output_node] + cell.outputs

Once this is in place, we need to add our engine as an entry point to our setup.py script - for this, see the following section.

Ensuring your engine is found by papermill

Custom engines can be specified as entry points, under the papermill.engine prefix. The entry point needs to reference the class that we have just implemented. For example, if you write an engine called TimingEngine in a package called papermill_timing, then in the setup.py file, you should specify:

from setuptools import setup, find_packages

setup(
    name="papermill_timing",
    version="0.1",
    url="https://github.com/my_username/papermill_timing.git",
    author="My Name",
    author_email="my.email@gmail.com",
    description="A papermill engine that logs additional timing information about code.",
    packages=find_packages("./src"),
    package_dir={"": "src"},
    install_requires=["papermill", "nbformat"],
    entry_points={"papermill.engine": ["timer_engine=papermill_timing:TimingEngine"]},
)

This allows users to specify the engine from papermill_timing by passing the command line argument --engine timer_engine.

In the image below, the notebook on the left was executed with the new custom engine, while the one on the left was executed with the standard papermill engine. As you can see, this adds our “injected” output to each code cell

Extending papermill by contributing to it

If you find that you’d like to not just add I/O and execution handlers, but think a fundamental aspect of the project could use some improvement, then you want contribute to it.

Development of papermill happens on github, and a detailed guide to contributing to it can be found there. There is also a code of conduct there. Please read both documents before beginning!

API Reference

If you are looking for information about a specific function, class, or method, this documentation section will help you.

Reference

This part of the documentation lists the full API reference of all public classes and functions.

CLI

papermill.cli

Main papermill interface.

papermill.cli.print_papermill_version(ctx, param, value)

Command Line options

Usage: papermill [OPTIONS] NOTEBOOK_PATH OUTPUT_PATH

  This utility executes a single notebook on a container.

  Papermill takes a source notebook, applies parameters to the source
  notebook, executes the notebook with the specified kernel, and saves the
  output in the destination notebook.

Options:
  -p, --parameters TEXT...        Parameters to pass to the parameters cell.
  -r, --parameters_raw TEXT...    Parameters to be read as raw string.
  -f, --parameters_file TEXT      Path to YAML file containing parameters.
  -y, --parameters_yaml TEXT      YAML string to be used as parameters.
  -b, --parameters_base64 TEXT    Base64 encoded YAML string as parameters.
  --inject-input-path             Insert the path of the input notebook as
                                  PAPERMILL_INPUT_PATH as a notebook
                                  parameter.
  --inject-output-path            Insert the path of the output notebook as
                                  PAPERMILL_OUTPUT_PATH as a notebook
                                  parameter.
  --inject-paths                  Insert the paths of input/output notebooks
                                  as
                                  PAPERMILL_INPUT_PATH/PAPERMILL_OUTPUT_PATH
                                  as notebook parameters.
  --engine TEXT                   The execution engine name to use in
                                  evaluating the notebook.
  --prepare-only / --prepare-execute
                                  Flag for outputting the notebook without
                                  execution, but with parameters applied.
  -k, --kernel TEXT               Name of kernel to run.
  --cwd TEXT                      Working directory to run notebook in.
  --progress-bar / --no-progress-bar
                                  Flag for turning on the progress bar.
  --log-output / --no-log-output  Flag for writing notebook output to stderr.
  --log-level [NOTSET|DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Set log level
  --start_timeout INTEGER         Time in seconds to wait for kernel to start.
  --report-mode / --not-report-mode
                                  Flag for hiding input.
  --version                       Flag for displaying the version.
  -h, --help                      Show this message and exit.

Workflow

papermill.engines

Engines to perform different roles

class papermill.engines.Engine

Bases: object

Base class for engines.

Other specific engine classes should inherit and implement the execute_managed_notebook method.

Defines execute_notebook method which is used to correctly setup the NotebookExecutionManager object for engines to interact against.

classmethod execute_managed_notebook(nb_man, kernel_name, **kwargs)

An abstract method where implementation will be defined in a subclass.

classmethod execute_notebook(nb, kernel_name, output_path=None, progress_bar=True, log_output=False, **kwargs)

A wrapper to handle notebook execution tasks.

Wraps the notebook object in a NotebookExecutionManager in order to track execution state in a uniform manner. This is meant to help simplify engine implementations. This allows a developer to just focus on iterating and executing the cell contents.

class papermill.engines.NBConvertEngine

Bases: papermill.engines.Engine

A notebook engine representing an nbconvert process.

This can execute a notebook document and update the nb_man.nb object with the results.

classmethod execute_managed_notebook(nb_man, kernel_name, log_output=False, start_timeout=60, execution_timeout=None, **kwargs)

Performs the actual execution of the parameterized notebook locally.

Parameters:

• nb (NotebookNode) – Executable notebook object.
• kernel_name (str) – Name of kernel to execute the notebook against.
• log_output (bool) – Flag for whether or not to write notebook output to stderr.
• start_timeout (int) – Duration to wait for kernel start-up.
• execution_timeout (int) – Duration to wait before failing execution (default: never).

Note: The preprocessor concept in this method is similar to what is used by nbconvert, and it is somewhat misleading here. The preprocesser represents a notebook processor, not a preparation object.

class papermill.engines.NotebookExecutionManager(nb, output_path=None, log_output=False, progress_bar=True)

Bases: object

Wrapper for execution state of a notebook.

This class is a wrapper for notebook objects to house execution state related to the notebook being run through an engine.

In particular the NotebookExecutionManager provides common update callbacks for use within engines to facilitate metadata and persistence actions in a shared manner.

COMPLETED = 'completed'

FAILED = 'failed'

PENDING = 'pending'

RUNNING = 'running'

cell_complete(cell, cell_index=None, **kwargs)

Finalize metadata for a cell and save notebook.

Optionally called by engines during execution to finalize the metadata for a cell and save the notebook to the output path.

cell_exception(cell, cell_index=None, **kwargs)

Set metadata when an exception is raised.

Called by engines when an exception is raised within a notebook to set the metadata on the notebook indicating the location of the failure.

cell_start(cell, cell_index=None, **kwargs)

Set and save a cell’s start state.

Optionally called by engines during execution to initialize the metadata for a cell and save the notebook to the output path.

cleanup_pbar()

Clean up a progress bar

complete_pbar()

Refresh progress bar

notebook_complete(**kwargs)

Finalize the metadata for a notebook and save the notebook to the output path.

Called by Engine when execution concludes, regardless of exceptions.

notebook_start(**kwargs)

Initialize a notebook, clearing its metadata, and save it.

When starting a notebook, this initializes and clears the metadata for the notebook and its cells, and saves the notebook to the given output path.

Called by Engine when execution begins.

now()

Helper to return current UTC time

save(**kwargs)

Saves the wrapped notebook state.

If an output path is known, this triggers a save of the wrapped notebook state to the provided path.

Can be used outside of cell state changes if execution is taking a long time to conclude but the notebook object should be synced.

For example, you may want to save the notebook every 10 minutes when running a 5 hour cell execution to capture output messages in the notebook.

set_timer()

Initializes the execution timer for the notebook.

This is called automatically when a NotebookExecutionManager is constructed.

class papermill.engines.PapermillEngines

Bases: object

The holder which houses any engine registered with the system.

This object is used in a singleton manner to save and load particular named Engine objects so they may be referenced externally.

execute_notebook_with_engine(engine_name, nb, kernel_name, **kwargs)

Fetch a named engine and execute the nb object against it.

get_engine(name=None)

Retrieves an engine by name.

register(name, engine)

Register a named engine

register_entry_points()

Register entrypoints for an engine

Load handlers provided by other packages

papermill.engines.catch_nb_assignment(func)

Wrapper to catch nb keyword arguments

This helps catch nb keyword arguments and assign onto self when passed to the wrapped function.

Used for callback methods when the caller may optionally have a new copy of the originally wrapped nb object.

papermill.execute

papermill.execute.execute_notebook(input_path, output_path, parameters=None, engine_name=None, prepare_only=False, kernel_name=None, progress_bar=True, log_output=False, start_timeout=60, report_mode=False, cwd=None)

Executes a single notebook locally.

Parameters:

• input_path (str) – Path to input notebook
• output_path (str) – Path to save executed notebook
• parameters (dict, optional) – Arbitrary keyword arguments to pass to the notebook parameters
• engine_name (str, optional) – Name of execution engine to use
• prepare_only (bool, optional) – Flag to determine if execution should occur or not
• kernel_name (str, optional) – Name of kernel to execute the notebook against
• progress_bar (bool, optional) – Flag for whether or not to show the progress bar.
• log_output (bool, optional) – Flag for whether or not to write notebook output_path to stderr
• start_timeout (int, optional) – Duration in seconds to wait for kernel start-up
• report_mode (bool, optional) – Flag for whether or not to hide input.
• cwd (str, optional) – Working directory to use when executing the notebook

Returns:

nb – Executed notebook object

Return type:

NotebookNode

papermill.execute.prepare_notebook_metadata(nb, input_path, output_path, report_mode=False)

Prepare metadata associated with a notebook and its cells

Parameters:

• nb (NotebookNode) – Executable notebook object
• input_path (str) – Path to input notebook
• output_path (str) – Path to write executed notebook
• report_mode (bool, optional) – Flag to set report mode

papermill.execute.raise_for_execution_errors(nb, output_path)

Assigned parameters into the appropriate place in the input notebook

Parameters:

• nb (NotebookNode) – Executable notebook object
• output_path (str) – Path to write executed notebook

papermill.preprocess

class papermill.preprocess.PapermillExecutePreprocessor(**kw)

Bases: nbconvert.preprocessors.execute.ExecutePreprocessor

Module containing a preprocessor that executes the code cells and updates outputs

log_output_message(output)

papermill_process(nb_man, resources)

This function acts as a replacement for the grandparent’s preprocess method.

We are doing this for the following reasons:

1. Notebooks will stop executing when they encounter a failure but not raise a CellException. This allows us to save the notebook with the traceback even though a CellExecutionError was encountered.
2. We want to write the notebook as cells are executed. We inject our logic for that here.
3. We want to include timing and execution status information with the metadata of each cell.

Parameters:

• nb_man (NotebookExecutionManager) – Engine wrapper of notebook being converted
• resources (dictionary) – Additional resources used in the conversion process. Allows preprocessors to pass variables into the Jinja engine.

preprocess(nb_man, resources, km=None)

Wraps the parent class process call slightly

process_message(*arg, **kwargs)

Processes a kernel message, updates cell state, and returns the resulting output object that was appended to cell.outputs.

The input argument cell is modified in-place.

Parameters:

• msg (dict) – The kernel message being processed.
• cell (nbformat.NotebookNode) – The cell which is currently being processed.
• cell_index (int) – The position of the cell within the notebook object.

Returns:

output – The execution output payload (or None for no output).

Return type:

dict

Raises:

CellExecutionComplete – Once a message arrives which indicates computation completeness.

Language Translators

Translators

Translator

class papermill.translators.Translator

classmethod assign(name, str_val)

classmethod codify(parameters)

classmethod comment(cmt_str)

classmethod translate(val)

Translate each of the standard json/yaml types to appropiate objects.

classmethod translate_bool(val)

Default behavior for translation

classmethod translate_dict(val)

classmethod translate_escaped_str(str_val)

Reusable by most interpreters

classmethod translate_float(val)

Default behavior for translation

classmethod translate_int(val)

Default behavior for translation

classmethod translate_list(val)

classmethod translate_none(val)

Default behavior for translation

classmethod translate_raw_str(val)

Reusable by most interpreters

classmethod translate_str(val)

Default behavior for translation

PapermillTranslators

class papermill.translators.PapermillTranslators

The holder which houses any translator registered with the system. This object is used in a singleton manner to save and load particular named Translator objects for reference externally.

find_translator(kernel_name, language)

register(language, translator)

Python

class papermill.translators.PythonTranslator

classmethod comment(cmt_str)

classmethod translate_bool(val)

Default behavior for translation

classmethod translate_dict(val)

classmethod translate_list(val)

R

class papermill.translators.RTranslator

classmethod comment(cmt_str)

classmethod translate_bool(val)

Default behavior for translation

classmethod translate_dict(val)

classmethod translate_list(val)

classmethod translate_none(val)

Default behavior for translation

Julia

class papermill.translators.JuliaTranslator

classmethod comment(cmt_str)

classmethod translate_dict(val)

classmethod translate_list(val)

classmethod translate_none(val)

Default behavior for translation

Scala

class papermill.translators.ScalaTranslator

classmethod assign(name, str_val)

classmethod comment(cmt_str)

classmethod translate_dict(val)

Translate dicts to scala Maps

classmethod translate_int(val)

Default behavior for translation

classmethod translate_list(val)

Translate list to scala Seq

Functions

papermill.translators.translate_parameters(kernel_name, language, parameters)

Input / Output

papermill.iorw

class papermill.iorw.ABSHandler

Bases: object

listdir(path)

pretty_path(path)

read(path)

write(buf, path)

class papermill.iorw.ADLHandler

Bases: object

listdir(path)

pretty_path(path)

read(path)

write(buf, path)

class papermill.iorw.GCSHandler

Bases: object

RATE_LIMIT_RETRIES = 3

RETRY_DELAY = 1

RETRY_MAX_DELAY = 4

RETRY_MULTIPLIER = 1

listdir(path)

pretty_path(path)

read(path)

write(buf, path)

class papermill.iorw.HttpHandler

Bases: object

classmethod listdir(path)

classmethod pretty_path(path)

classmethod read(path)

classmethod write(buf, path)

class papermill.iorw.LocalHandler

Bases: object

cwd(new_path)

Sets the cwd during reads and writes

listdir(path)

pretty_path(path)

read(path)

write(buf, path)

class papermill.iorw.PapermillIO

Bases: object

The holder which houses any io system registered with the system. This object is used in a singleton manner to save and load particular named Handler objects for reference externally.

get_handler(path)

listdir(path)

pretty_path(path)

read(path, extensions=['.ipynb', '.json'])

register(scheme, handler)

register_entry_points()

reset()

write(buf, path, extensions=['.ipynb', '.json'])

class papermill.iorw.S3Handler

Bases: object

classmethod listdir(path)

classmethod pretty_path(path)

classmethod read(path)

classmethod write(buf, path)

papermill.iorw.get_pretty_path(path)

papermill.iorw.list_notebook_files(path)

Returns a list of all the notebook files in a directory.

papermill.iorw.load_notebook_node(notebook_path)

Returns a notebook object with papermill metadata loaded from the specified path.

Parameters:notebook_path (str) – Path to the notebook file. Returns:nbformat.NotebookNode

papermill.iorw.local_file_io_cwd(path=None)

papermill.iorw.read_yaml_file(path)

Reads a YAML file from the location specified at ‘path’.

papermill.iorw.write_ipynb(nb, path)

Saves a notebook object to the specified path. :param nb_node: Notebook object to save. :type nb_node: nbformat.NotebookNode :param notebook_path: Path to save the notebook object to. :type notebook_path: str

Storage

Azure

These modules outline how to interact with Azure data stores, specifically Azure Blob Storage and Azure Data Lakes.

papermill.abs module

papermill.adl module

AWS

This module shows how to interact with AWS S3 data stores.

papermill.s3 module

Utilities

Utils

papermill.utils.chdir(path)

Change working directory to path and restore old path on exit.

path can be None in which case this is a no-op.

papermill.utils.retry(num)

Exceptions

exception papermill.exceptions.AwsError

Raised when an AWS Exception is encountered.

exception papermill.exceptions.FileExistsError

Raised when a File already exists on S3.

exception papermill.exceptions.PapermillException

Raised when an exception is encountered when operating on a notebook.

exception papermill.exceptions.PapermillExecutionError(exec_count, source, ename, evalue, traceback)

Raised when an exception is encountered in a notebook.

exception papermill.exceptions.PapermillMissingParameterException

Raised when a parameter without a value is required to operate on a notebook.

exception papermill.exceptions.PapermillOptionalDependencyException

Raised when an exception is encountered when an optional plugin is missing.

exception papermill.exceptions.PapermillRateLimitException

Raised when an io request has been rate limited

papermill.exceptions.missing_dependency_generator(package, dep)

papermill.exceptions.missing_environment_variable_generator(package, env_key)

Log

Sets up a logger

papermill.tests package

Submodules

papermill.tests.test_abs module

papermill.tests.test_adl module

papermill.tests.test_cli module

papermill.tests.test_conf module

papermill.tests.test_engines module

papermill.tests.test_engines.AnyMock(cls)

Mocks a matcher for any instance of class cls. e.g. my_mock.called_once_with(Any(int), “bar”)

class papermill.tests.test_engines.TestEngineBase(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

test_cell_callback_execute()

test_no_cell_callback_execute()

test_wrap_and_execute_notebook()

Mocks each wrapped call and proves the correct inputs get applies to the correct underlying calls for execute_notebook.

class papermill.tests.test_engines.TestEngineRegistration(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

test_getting()

test_registering_entry_points()

test_registration()

class papermill.tests.test_engines.TestNBConvertEngine(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

test_nb_convert_engine()

test_nb_convert_engine_execute()

test_nb_convert_log_outputs()

test_nb_convert_no_log_outputs()

class papermill.tests.test_engines.TestNotebookExecutionManager(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

test_basic_pbar()

test_cell_complete_after_cell_exception()

test_cell_complete_after_cell_start()

test_cell_complete_new_nb()

test_cell_complete_without_cell_start()

test_cell_exception()

test_cell_exception_new_nb()

test_cell_start()

test_cell_start_new_nb()

test_nb_isolation()

Tests that the engine notebook is isolated from source notebook

test_no_pbar()

test_notebook_complete()

test_notebook_complete_cell_status_completed()

test_notebook_complete_cell_status_with_failed()

test_notebook_complete_new_nb()

test_notebook_start()

test_notebook_start_new_nb()

test_save()

test_save_new_nb()

test_save_no_output()

test_set_timer()

papermill.tests.test_exceptions module

papermill.tests.test_execute module

class papermill.tests.test_execute.TestBrokenNotebook1(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test()

class papermill.tests.test_execute.TestBrokenNotebook2(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test()

class papermill.tests.test_execute.TestCWD(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test_execution_respects_cwd_assignment()

test_local_save_ignores_cwd_assignment()

class papermill.tests.test_execute.TestNBConvertCalls(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

test_convert_output_to_html()

class papermill.tests.test_execute.TestNotebookHelpers(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test_backslash_params()

test_backslash_quote_params()

test_cell_insertion()

test_default_start_timeout(preproc_mock)

test_double_backslash_quote_params()

test_no_tags()

test_prepare_only()

test_quoted_params()

test_start_timeout(preproc_mock)

class papermill.tests.test_execute.TestReportMode(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown()

Hook method for deconstructing the test fixture after testing it.

test_report_mode()

papermill.tests.test_execute.execute_notebook(input_path, output_path, parameters=None, engine_name=None, prepare_only=False, *, kernel_name='python3', progress_bar=True, log_output=False, start_timeout=60, report_mode=False, cwd=None)

Executes a single notebook locally.

Parameters:

• input_path (str) – Path to input notebook
• output_path (str) – Path to save executed notebook
• parameters (dict, optional) – Arbitrary keyword arguments to pass to the notebook parameters
• engine_name (str, optional) – Name of execution engine to use
• prepare_only (bool, optional) – Flag to determine if execution should occur or not
• kernel_name (str, optional) – Name of kernel to execute the notebook against
• progress_bar (bool, optional) – Flag for whether or not to show the progress bar.
• log_output (bool, optional) – Flag for whether or not to write notebook output_path to stderr
• start_timeout (int, optional) – Duration in seconds to wait for kernel start-up
• report_mode (bool, optional) – Flag for whether or not to hide input.
• cwd (str, optional) – Working directory to use when executing the notebook

Returns:

nb – Executed notebook object

Return type:

NotebookNode

papermill.tests.test_gcs module

class papermill.tests.test_gcs.GCSTest(methodName='runTest')

Bases: unittest.case.TestCase

Tests for GCS.

setUp()

Hook method for setting up the test fixture before exercising it.

test_gcs_handle_exception(mock_gcs_filesystem)

test_gcs_invalid_code(mock_gcs_filesystem)

test_gcs_listdir(mock_gcs_filesystem)

test_gcs_read(mock_gcs_filesystem)

test_gcs_retry(mock_gcs_filesystem)

test_gcs_retry_older_exception(mock_gcs_filesystem)

test_gcs_retry_unknown_failure_code(mock_gcs_filesystem)

test_gcs_unretryable(mock_gcs_filesystem)

test_gcs_write(mock_gcs_filesystem)

class papermill.tests.test_gcs.MockGCSFile(exception=None, max_raises=1)

Bases: object

read()

write(buf)

papermill.tests.test_gcs.mock_gcs_fs_wrapper(exception=None, max_raises=1)

papermill.tests.test_iorw module

papermill.tests.test_parameterize module

class papermill.tests.test_parameterize.TestBuiltinParameters(methodName='runTest')

Bases: unittest.case.TestCase

test_add_builtin_parameters_adds_dict_of_builtins()

test_add_builtin_parameters_allows_to_override_builtin()

test_add_builtin_parameters_keeps_provided_parameters()

test_builtin_parameters_include_current_datetime_local()

test_builtin_parameters_include_current_datetime_utc()

test_builtin_parameters_include_run_uuid()

class papermill.tests.test_parameterize.TestNotebookParametrizing(methodName='runTest')

Bases: unittest.case.TestCase

count_nb_injected_parameter_cells(nb)

test_injected_parameters_tag()

test_no_parameter_tag()

test_no_tag_copying()

test_repeated_run_injected_parameters_tag()

test_repeated_run_no_parameters_tag()

class papermill.tests.test_parameterize.TestPathParameterizing(methodName='runTest')

Bases: unittest.case.TestCase

test_parameterized_path_with_none_parameters()

test_parameterized_path_with_undefined_parameter()

test_path_with_boolean_parameter()

test_path_with_dict_parameter()

test_path_with_float_format_string()

test_path_with_list_parameter()

test_path_with_multiple_parameter()

test_path_with_none_parameter()

test_path_with_numeric_format_string()

test_path_with_numeric_parameter()

test_path_with_single_parameter()

test_plain_text_path_with_empty_parameters_object()

test_plain_text_path_with_none_parameters()

test_plain_text_path_with_unused_parameters()

papermill.tests.test_preprocessor module

class papermill.tests.test_preprocessor.TestPapermillExecutePreprocessor(methodName='runTest')

Bases: unittest.case.TestCase

setUp()

Hook method for setting up the test fixture before exercising it.

test_logging_data_msg()

test_logging_stderr_msg()

test_logging_stdout_msg()

papermill.tests.test_s3 module

papermill.tests.test_translators module

papermill.tests.test_utils module

Module contents

papermill.tests.get_notebook_dir(*args)

papermill.tests.get_notebook_path(*args)

Indices and tables

• Index
• Module Index
• Search Page