Interface of policy


About policy

Policies give you fine-grained control over the training process. This example demonstrates how policies work and how you can create your own policies.

Parameter Spaces and Phases

Parameter spaces like linspace and cosinespace are the basic building blocks.

from poutyne import linspace, cosinespace

You can define the space and iterate over them:

space = linspace(1, 0, 3)
for i in space:
space = cosinespace(1, 0, 5)
for i in space:

You can use the space and create a phase with them:

from poutyne import Phase

phase = Phase(lr=linspace(0, 1, 3))

# and iterate
for d in phase:

You can also visualize your phase:

import matplotlib.pyplot as plt

Phases can have multiple parameters:

phase = Phase(
    lr=linspace(0, 1, 10),
    momentum=cosinespace(.99, .9, 10),


Visualize Different Phases

steps = 100

fig, ax = plt.subplots()
# Constant value
Phase(lr=linspace(.7, .7, steps)).plot(ax=ax)
# Linear
Phase(lr=linspace(0, 1, steps)).plot(ax=ax)
# Cosine
Phase(lr=cosinespace(1, 0, steps)).plot(ax=ax);

Visualize Multiple Parameters in One Phase

steps = 100
phase = Phase(lr=linspace(1, 0.5, steps), momentum=cosinespace(.8, 1, steps))

fig, axes = plt.subplots(1, 2, figsize=(12, 3))
phase.plot("lr", ax=axes[0])
phase.plot("momentum", ax=axes[1]);

Build Complex Policies From Basic Phases

You can build complex optimizer policies by chaining phases together:

from poutyne import OptimizerPolicy

policy = OptimizerPolicy([
    Phase(lr=linspace(0, 1, 100)),
    Phase(lr=cosinespace(1, 0, 200)),
    Phase(lr=linspace(0, .5, 100)),
    Phase(lr=linspace(.5, .1, 300)),


Use Already Defined Complex Policies

It’s easy to build your own policies, but Poutyne contains some pre-defined phases.

from poutyne import sgdr_phases

# build them manually
policy = OptimizerPolicy([
    Phase(lr=cosinespace(1, 0, 200)),
    Phase(lr=cosinespace(1, 0, 400)),
    Phase(lr=cosinespace(1, 0, 800)),

# or use the pre-defined one
policy = OptimizerPolicy(sgdr_phases(base_cycle_length=200, cycles=3, cycle_mult=2))

Pre-defined ones are just a list phases:

sgdr_phases(base_cycle_length=200, cycles=3, cycle_mult=2)

Here is the one-cycle policy:

from poutyne import one_cycle_phases

tp = OptimizerPolicy(one_cycle_phases(steps=500))