PAIConfig
class perforatedai.PAIConfig() Configuration class for Perforated AI (PAI) settings.
This class manages all configuration parameters for the Perforated AI system, including device settings, dendrite behavior, module conversion rules, training parameters, and debugging options.
Parameters
No parameters (all settings have default values and are configured via setter methods after initialization)
Attributes
Device Settings
use_cuda (bool) – Whether CUDA is available and should be used. (default: True if CUDA available, else False)
device (torch.device) – The device to use for computation (CPU, CUDA, etc.). (default: cuda if available, else cpu)
Debug Settings
debugging_input_dimensions (int) – Debug level for input dimension checking. Should only be set to 0 (not debugging) or 1 (initialize debugging). (default: 0)
confirm_correct_sizes (bool) – Whether to verify tensor sizes during execution. This will slow things down slightly but can help catch dimension errors. (default: False)
verbose (bool) – Enable verbose logging output. (default: False)
extra_verbose (bool) – Enable extra verbose logging output. (default: False)
silent (bool) – Suppress all PAI print statements. (default: False)
Confirmation Flags
unwrapped_modules_confirmed (bool) – Confirmation flag for using unwrapped modules (not recommended). (default: False)
weight_decay_accepted (bool) – Confirmation flag for accepting weight decay. (default: False)
checked_skipped_modules (bool) – Whether skipped modules have been verified. (default: False)
File and Visualization Settings
save_name (str) – Name used for saving models. Should not be set manually. (default: "PAI")
save_old_graph_scores (bool) – Whether to save historical graph scores. (default: True)
using_safe_tensors (bool) – Use safe tensors file format for saving. (default: True)
drawing_pai (bool) – Enable PAI visualization graphs. Can be set to False for custom training visualizations. (default: True)
test_saves (bool) – Save intermediary test models. Good for experimentation, uses more memory. (default: True)
pai_saves (bool) – Save PAI-specific format models with slight memory and speed improvements. (default: False)
Input Configuration
input_dimensions (list of int) – Format specification for input tensor dimensions. Set neuron index to 0, variable indexes to -1. For example, for format [batch_size, nodes, x, y] use [-1, 0, -1, -1]. For [batch_size, time_index, nodes] use [-1, -1, 0]. (default: [-1, 0, -1, -1])
Dendrite Settings
global_candidates (int) – Number of global candidate dendrites. Reserved for future implementation options. (default: 1)
testing_dendrite_capacity (bool) – Enable dendrite capacity testing mode. (default: True)
candidate_weight_initialization_multiplier (float) – Multiplier for random dendrite weight initialization. (default: 0.01)
retain_all_dendrites (bool) – Keep dendrites even if they don't improve performance. (default: False)
max_dendrite_tries (int) – Maximum attempts to add dendrites with random initializations. Set to 1 to quit after first failure, higher values try new random weights. (default: 2)
max_dendrites (int) – Maximum total number of dendrites to add even if they continue improving scores. (default: 100)
d_type (torch.dtype) – Data type for dendrite weights. (default: torch.float)
pai_forward_function (callable) – Activation function used for dendrites. (default: torch.sigmoid)
Improvement Thresholds
improvement_threshold (float) – Relative improvement threshold (percentage) needed to declare a new best validation score. (default: 0.0001)
improvement_threshold_raw (float) – Absolute improvement threshold needed to declare a new best validation score. (default: 1e-5)
Switch Mode Settings
switch_mode (int) – Current switch mode setting. Controls when dendrites are added during training. Options:
- • DOING_SWITCH_EVERY_TIME (0): Add dendrites every epoch (for debugging)
- • DOING_HISTORY (1): Add dendrites when validation hasn't improved (default)
- • DOING_FIXED_SWITCH (2): Add dendrites at fixed intervals
- • DOING_NO_SWITCH (3): Never add dendrites (regular training)
(default: 1)
n_epochs_to_switch (int) – Number of epochs without improvement before adding dendrites. Should be higher than scheduler patience. Only used when switch_mode is DOING_HISTORY. (default: 10)
history_lookback (int) – Number of epochs to average for validation history. (default: 1)
initial_history_after_switches (int) – Number of epochs to wait after adding dendrites before checking to add more. (default: 0)
fixed_switch_num (int) – Number of epochs between fixed switches. Only used when switch_mode is DOING_FIXED_SWITCH. (default: 250)
first_fixed_switch_num (int) – Number of epochs before first switch for initial pretraining. Only used when switch_mode is DOING_FIXED_SWITCH. (default: 249)
reset_best_score_on_switch (bool) – Whether to reset best score when adding dendrites. Useful if many epochs are needed to catch up to previous best score. (default: False)
Learning Rate Management
find_best_lr (bool) – Automatically sweep over previously used learning rates when adding new dendrites. Sometimes best to return to initial LR, often best to start at lower LR. (default: True)
dont_give_up_unless_learning_rate_lowered (bool) – Ensure learning rate search lowers LR at least once even if previous epoch didn't lower it. (default: True)
Scheduler Parameters
param_vals_setting (int) – Current parameter tracking mode for scheduler. Options:
- • PARAM_VALS_BY_TOTAL_EPOCH (0): Track by total epochs
- • PARAM_VALS_BY_UPDATE_EPOCH (1): Reset params at every switch (default)
- • PARAM_VALS_BY_NEURON_EPOCH_START (2): Reset for dendrite starts only (not used in open source)
(default: 1)
Advanced Settings
learn_dendrites_live (bool) – Enable live dendrite learning. Not used in open source implementation, leave as default. (default: False)
no_extra_n_modes (bool) – Disable extra neuron modes. Not used in open source implementation, leave as default. (default: True)
perforated_backpropagation (bool) – Whether Perforated BackpropagationTM is enabled. Set automatically if library is available. (default: False)
Module Conversion Settings
modules_to_convert (list) – Module types to convert to PAI modules. (default: [nn.Conv1d, nn.Conv2d, nn.Conv3d, nn.Linear])
module_names_to_convert (list of str) – Module names to convert to PAI modules. (default: ["PAISequential"])
module_ids_to_convert (list of str) – Specific module IDs to convert to PAI modules by assigned ID (e.g., "model.conv2"). (default: [])
modules_to_track (list) – Module types to track but not convert. All modules should either be converted or tracked. (default: [])
module_names_to_track (list of str) – Module names to track but not convert. (default: [])
module_ids_to_track (list of str) – Specific module IDs to track but not convert. (default: [])
modules_to_replace (list) – Module types to replace before conversion. Replacement happens before conversion, so replaced modules are then run through conversion steps. (default: [])
replacement_modules (list) – Replacement modules for modules_to_replace. (default: [])
modules_with_processing (list) – Module types requiring custom processing. For modules that are not one tensor input and one tensor output. (default: [])
modules_processing_classes (list) – Processing classes for modules_with_processing. (default: [])
module_names_with_processing (list of str) – Module names requiring custom processing. (default: [])
module_by_name_processing_classes (list) – Processing classes for module_names_with_processing. (default: [])
module_names_to_not_save (list of str) – Module names to exclude from saving. Some models have multiple pointers to the same modules which cause problems. (default: [".base_model"])
Examples
>>> import perforatedai as pai
>>> config = pai.PAIConfig()
>>> config.set_verbose(True)
>>> config.set_switch_mode(config.DOING_FIXED_SWITCH)
>>> config.set_fixed_switch_num(100)
>>> config.set_input_dimensions([-1, 0, -1, -1]) # [batch, neurons, height, width]PAISequential
class perforatedai.PAISequential(layer_array) Sequential module wrapper for PAI.
This wrapper takes an array of layers and creates a sequential container that is compatible with PAI's dendrite addition system. It should be used for normalization layers and can be used for final output layers.
Parameters
layer_array (list) – List of PyTorch nn.Module objects to be executed sequentially.
Notes
This should be used for:
- • All normalization layers (LayerNorm, BatchNorm, etc.)
This can be used for:
- • Final output layer and softmax combinations
Examples
>>> import torch.nn as nn
>>> from perforatedai import PAISequential
>>>
>>> hidden_dim = 128
>>> seq_width = 256
>>> layers = [nn.Linear(2 * hidden_dim, seq_width),
... nn.LayerNorm(seq_width)]
>>> sequential_block = PAISequential(layers)Methods
forward
**forward(*args, kwargs) Forward pass through the sequential layers.
Parameters:
- • *args – Positional arguments passed to the first layer
- • **kwargs – Keyword arguments passed to the layers
Returns:
torch.Tensor – Output from the final layer in the sequence
add_pbp_var
perforatedai.add_pbp_var(obj, var_name, initial_value) Dynamically add a property with getter and setter to an object.
This function adds a private variable along with getter and setter methods to a given object instance. Used for integrating Perforated BackpropagationTM variables into the PAIConfig class.
Parameters
obj (object) – The object to which the property will be added.
var_name (str) – Name of the variable/property to create.
initial_value (any) – Initial value for the property.
Returns
None
Notes
Creates three attributes on obj:
- • _{var_name}: private storage
- • get_{var_name}: getter method
- • set_{var_name}: setter method
This function is automatically called when Perforated BackpropagationTM library is available and should not typically be called manually by users.