objax.privacy.dpsgd package

PrivateGradValues(f, vc, noise_multiplier, ...)

Computes differentially private gradients as required by DP-SGD.

analyze_dp(q, noise_multiplier, steps[, ...])

Compute and print results of DP-SGD analysis.

analyze_renyi(q, noise_multiplier, steps, orders)

Compute RDP of the Sampled Gaussian Mechanism.

convert_renyidp_to_dp(orders, rdp[, ...])

Compute delta (or eps) for given eps (or delta) from RDP values.

class objax.privacy.dpsgd.PrivateGradValues(f, vc, noise_multiplier, l2_norm_clip, microbatch, batch_axis=(0,), keygen=objax.random.Generator(seed=0), use_norm_accumulation=False)[source]

Computes differentially private gradients as required by DP-SGD. This module can be used in place of GradVals, and automatically makes the optimizer differentially private.

Parameters
__init__(f, vc, noise_multiplier, l2_norm_clip, microbatch, batch_axis=(0,), keygen=objax.random.Generator(seed=0), use_norm_accumulation=False)[source]

Constructs a PrivateGradValues instance.

Parameters
  • f (Callable) – the function for which to compute gradients.

  • vc (objax.variable.VarCollection) – the variables for which to compute gradients.

  • noise_multiplier (float) – scale of standard deviation for added noise in DP-SGD.

  • l2_norm_clip (float) – value of clipping norm for DP-SGD.

  • microbatch (int) – the size of each microbatch.

  • batch_axis (Tuple[Optional[int], ...]) – the axis to use as batch during vectorization. Should be a tuple of 0s.

  • keygen (objax.random.random.Generator) – a Generator for random numbers. Defaults to objax.random.DEFAULT_GENERATOR.

  • use_norm_accumulation (bool) – whether to use norm accumulation as a first step to compute the clipped gradients (more details further). This is more memory efficient and often faster when the model is very deep and uses a large batch-size.

reshape_microbatch(x)[source]

Reshapes examples into microbatches. DP-SGD requires that per-example gradients are clipped and noised, however this can be inefficient. To speed this up, it is possible to clip and noise a microbatch of examples, at a sight cost to privacy. If speed is not an issue, the microbatch size should be set to 1.

If x has shape [D0, D1, …, Dn], the reshaped output will have shape [number_of_microbatches, microbatch_size, D1, …, DN].

Parameters

x (Union[jax._src.numpy.lax_numpy.ndarray, jaxlib.xla_extension.DeviceArrayBase, jaxlib.xla_extension.pmap_lib.ShardedDeviceArrayBase]) – items to be reshaped.

Returns

The reshaped items.

Return type

Union[jax._src.numpy.lax_numpy.ndarray, jaxlib.xla_extension.DeviceArrayBase, jaxlib.xla_extension.pmap_lib.ShardedDeviceArrayBase]

__call__(*args)[source]

Returns the computed DP-SGD gradients.

Returns

A tuple (gradients, value of f).

objax.privacy.dpsgd.analyze_dp(q, noise_multiplier, steps, orders=(1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20), delta=1e-05)[source]

Compute and print results of DP-SGD analysis.

Parameters
  • q (float) – The sampling rate.

  • noise_multiplier (float) – The ratio of the standard deviation of the Gaussian noise to the l2-sensitivity of the function to which it is added.

  • steps (int) – The number of steps.

  • orders (Tuple[float, ...]) – An array (or a scalar) of RDP orders.

  • delta (float) – The target delta.

Returns

eps

Raises

ValueError – If target_delta are messed up.

Return type

float

objax.privacy.dpsgd.analyze_dp_sample_without_replacement(q, noise_multiplier, steps, orders=(1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 3.0, 3.5, 4.0, 4.5, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20), delta=1e-05)[source]

Compute and print results of DP-SGD analysis for sample without replacement.

Parameters
  • q (float) – The sampling rate.

  • noise_multiplier (float) – The ratio of the standard deviation of the Gaussian noise to the l2-sensitivity of the function to which it is added.

  • steps (int) – The number of steps.

  • orders (Tuple[float, ...]) – An array (or a scalar) of RDP orders.

  • delta (float) – The target delta.

Returns

eps

Raises

ValueError – If target_delta are messed up.

Return type

float

objax.privacy.dpsgd.analyze_renyi(q, noise_multiplier, steps, orders)[source]

Compute RDP of the Sampled Gaussian Mechanism.

Parameters
  • q (float) – The sampling rate.

  • noise_multiplier (float) – The ratio of the standard deviation of the Gaussian noise to the l2-sensitivity of the function to which it is added.

  • steps (int) – The number of steps.

  • orders (Union[float, Tuple[float, ...]]) – An array (or a scalar) of RDP orders.

Returns

The RDPs at all orders. Can be np.inf.

objax.privacy.dpsgd.analyze_renyi_sample_without_replacement(q, noise_multiplier, steps, orders)[source]

Compute RDP of Gaussian Mechanism using sampling without replacement.

This function applies to the following schemes: 1. Sampling w/o replacement: Sample a uniformly random subset of size m = q*n. 2. ``Replace one data point’’ version of differential privacy, i.e., n is

considered public information.

Reference: Theorem 27 of https://arxiv.org/pdf/1808.00087.pdf (A strengthened version applies subsampled-Gaussian mechanism) - Wang, Balle, Kasiviswanathan. “Subsampled Renyi Differential Privacy and Analytical Moments Accountant.” AISTATS’2019.

Parameters
  • q (float) – The sampling proportion = m / n. Assume m is an integer <= n.

  • noise_multiplier (float) – The ratio of the standard deviation of the Gaussian noise to the l2-sensitivity of the function to which it is added.

  • steps (int) – The number of steps.

  • orders (Union[float, Tuple[float, ...]]) – An array (or a scalar) of RDP orders.

Returns

The RDPs at all orders, can be np.inf.

Return type

float

objax.privacy.dpsgd.convert_renyidp_to_dp(orders, rdp, target_eps=None, target_delta=None)[source]

Compute delta (or eps) for given eps (or delta) from RDP values.

Parameters
  • orders (Tuple[float, ...]) – An array (or a scalar) of RDP orders.

  • rdp (Tuple[float, ...]) – An array of RDP values. Must be of the same length as the orders list.

  • target_eps (Optional[float]) – If not None, the epsilon for which we compute the corresponding delta.

  • target_delta (Optional[float]) – If not None, the delta for which we compute the corresponding epsilon. Exactly one of target_eps and target_delta must be None.

Returns

eps, delta, opt_order.

Raises

ValueError – If target_eps and target_delta are messed up.

Return type

Tuple[float, float, float]