From Sound to Tensors

Overview

An audio model receives numbers arranged into tensors. Before that point, the recording passes through several representations: a physical signal, sampled values, file bytes, an array, and finally a batch. Each conversion changes a specific property, such as the dtype, amplitude scale, sample rate, or axis order.

These notes follow that sequence through the Audio Tensor Lab pipeline. The examples use PCM16 WAV input and a small waveform classifier. Shapes such as [B, C, T] describe this pipeline's input convention rather than a universal format for every audio model.

sound → microphone → sampling → PCM / WAV
      → NumPy → float32 → mono → resampling
      → PyTorch tensor → padded batch → model → loss → gradients

Shape notation used here:

Index

1. Sound becomes digital data
2. Samples and sample rate
3. PCM and WAV
4. WAV bytes become NumPy arrays
5. PCM16 becomes float32
6. Stereo becomes mono
7. Resampling
8. NumPy becomes a PyTorch tensor
9. Batching variable-length audio
10. What the model receives
11. From logits to gradients

Sound becomes digital data

Speech produces changes in air pressure. A microphone converts those changes into an electrical signal whose amplitude varies over time. At this stage, the signal is continuous: it has not yet become a list of values that a program can read.

An analogue-to-digital converter measures the signal at regularly spaced instants. Each measurement becomes a numerical sample. The stored sequence represents how the signal changes over time; these samples are the starting point for the rest of the audio pipeline.

• Sound: changes in air pressure over time.
• Microphone: converts pressure changes into an electrical signal.
• Sampling: measures that signal at discrete time intervals.
• Digital audio: stores the measurements as numbers.

air-pressure changes → electrical signal → measurements → numbers

Samples and sample rate

A sequence such as 120, 340, 510, 200, -100, -420 records successive amplitude measurements. The values describe the waveform, but the sequence alone does not say how much time elapsed between measurements. The sample rate supplies that time scale.

For multi-channel audio, measurements from the same instant form a frame. A stereo frame contains a left-channel sample and a right-channel sample. Counting frames keeps duration calculations independent of the number of channels.

16 kHz = 16,000 frames per second
48 kHz = 48,000 frames per second

duration in seconds = number of frames / sample rate

Examples:

• Mono: 48,000 frames / 16,000 Hz = 3 seconds.
• Stereo at 48 kHz: one second contains 48,000 frames × 2 channels = 96,000 sample values.
• Sample count needs a sample rate to have a duration.

PCM and WAV

PCM describes how waveform samples are represented numerically. WAV describes how audio data and its metadata are packaged in a file. In a PCM16 WAV file, the audio payload contains signed 16-bit sample values, while the container provides the information needed to read them correctly.

That distinction matters during decoding. A byte buffer has no built-in notion of channels or time. Reading the same bytes with the wrong sample width or byte order produces different numbers. Using the wrong channel count groups those numbers incorrectly, and using the wrong sample rate gives them the wrong duration.

• PCM (Pulse Code Modulation): numerical representation of waveform samples.
• PCM16: signed 16-bit samples; range −32768 to 32767; two bytes per sample.
• WAV: container holding audio data and the metadata needed to interpret it. This pipeline uses PCM16 WAV.

WAV
├── metadata
│   ├── sample rate
│   ├── channels
│   ├── sample width
│   └── number of frames
└── PCM audio bytes

Decoding depends on sample width, byte order, channel count, and sample rate. Raw bytes alone do not specify these.

WAV bytes become NumPy arrays

Reading audio frames from a PCM WAV file yields raw bytes. NumPy interprets those bytes using a dtype that specifies how many bytes belong to each sample and how to decode them. For PCM16, each pair of bytes represents one signed integer. This step interprets the stored values; it does not yet normalize their amplitude or change the sample rate.

For a little-endian PCM16 buffer:

samples = np.frombuffer(raw_bytes, dtype="<i2")

• Mono buffer: shape [T].
• Stereo buffer: interleaved values L0, R0, L1, R1, ….
• Reshape multi-channel data to make the channel axis explicit:

samples = samples.reshape(num_frames, channels)

Example: [48000, 2] = 48,000 frames, two channels.

Reshaping groups each left/right pair into one row. The values remain in the same sequence, but the array now exposes separate time and channel axes. That makes operations such as selecting a channel or averaging channels explicit.

A view can share the underlying storage instead of copying it. In an interleaved stereo array, adjacent left-channel values are separated by a right-channel value in memory. Selecting samples[:, 0] therefore does not necessarily produce a contiguous block, even though it looks like a simple one-dimensional array. Strides describe how the array moves through that storage.

Memory notes:

• An array has a dtype, item size, shape, and strides.
• samples[:, 0] selects the first channel.
• That slice can be a non-contiguous view sharing the original storage.

PCM16 becomes float32

PCM16 is compact for storage. Neural-network computation typically uses floating-point inputs, so this pipeline converts the samples to float32 and scales the signed integer range to approximately [-1, 1].

Dividing by 32768 maps the most negative PCM16 value to exactly -1.0. The largest positive value is 32767, so its scaled value is slightly below 1.0. This scaling changes the numerical representation of the waveform while preserving the sequence of time positions and the relative amplitudes.

Convert the integer samples to floating point and scale their amplitude:

samples = samples.astype(np.float32) / 32768.0

• Cast happens before division.
• Sample count and sample rate stay the same.
• Storage per value: int16 = two bytes; float32 = four bytes.
• The resulting array uses roughly twice the raw storage.

The memory difference comes from the dtype, not from a change in duration. An array with the same shape contains the same number of values, but each value now occupies four bytes instead of two. This distinction between numerical precision and storage size also appears later in model computation with formats such as FP32, FP16, BF16, and INT8.

Stereo becomes mono

A stereo recording has two amplitude values at every time position. When the model expects mono input, those channels must be combined into one waveform. A simple conversion takes the mean of the channel values at each frame.

The axis is part of the operation's meaning. In [T, C], each row is a time position and each column is a channel. Averaging across columns produces one value per row, preserving the time sequence. Averaging across rows instead produces one value per channel and removes that sequence.

For an input shaped [T, C], average across channels:

mono = samples.astype(np.float32).mean(axis=1)

[T, C] → mean(axis=1) → [T]
 ↑  ↑
 0  1

• axis=1: combines channels; preserves time.
• axis=0: averages across time; removes the waveform's time axis.
• The float cast makes the channel averaging a floating-point operation.
• Mono conversion applies when the model expects a single channel.

Resampling

Purpose: convert audio to the sample rate expected by the model.

A recording may arrive at 48 kHz while the model expects 16 kHz. The target rate determines how many time positions represent one second of audio. Resampling constructs a representation on that new time grid, changing the number of frames while keeping the recording's duration approximately the same.

Example: three seconds of audio, 48 kHz → 16 kHz.

Before: 48,000 frames/sec × 3 sec = 144,000 frames
After:  16,000 frames/sec × 3 sec =  48,000 frames

• Sample rate changes.
• Frame count changes proportionally.
• Duration stays approximately constant.

Aliasing

Reducing the sample rate also reduces the frequency range that can be represented. A component above the new Nyquist limit cannot retain its original frequency on the lower-rate grid. Without filtering, it can appear as a lower frequency in the resulting signal. This distortion is called aliasing.

For 48 kHz → 16 kHz conversion, taking every third sample gives the expected output length. However, the length alone does not establish that the resulting waveform is correct. Frequencies above the new limit need to be filtered before samples are dropped.

• Nyquist limit: half the sample rate.
• At 48 kHz: about 24 kHz.
• At 16 kHz: about 8 kHz.
• Frequencies above the new limit can fold into lower frequencies during downsampling.
• An anti-aliasing low-pass filter removes those frequencies before reducing the sampling rate.

Common mix-ups

Example: the same 48,000 frames last one second at 48 kHz, but three seconds at 16 kHz.

Changing only the sample-rate metadata leaves the values and their count unchanged. It tells the reader to play the same sequence at a different speed, which explains the duration and pitch change. Resampling instead calculates sample values on the target grid.

Implementation used in Audio Tensor Lab: scipy.signal.resample_poly.

NumPy becomes a PyTorch tensor

The preprocessing arrays use time-first shapes: [T] for mono and [T, C] for multiple channels. The classifier uses channel-first waveforms, so this conversion also establishes the axis convention used by the model.

A mono waveform gains a channel axis, becoming [1, T]. A multi-channel waveform changes axis order from [T, C] to [C, T]. Adding the batch axis then produces [B, C, T]. These transformations preserve the meaning of time and channels while putting them in the positions expected by the next stage.

An incorrect axis order can make the model treat time positions as channels. The tensor may still contain all the original values, but their interpretation is wrong. Tracking the meaning of each axis is therefore as necessary as tracking its size.

Shape conventions in this pipeline:

• Mono needs a channel dimension.
• Multi-channel input switches from time-first to channel-first.
• Conv1d uses [B, C, T] for batched input.
• Passing [T, C] where [C, T] is expected swaps the meaning of the axes.

Batching variable-length audio

Recordings often have different durations. After conversion to a common sample rate, those durations still produce different values of T. A dense batch requires a rectangular shape, so waveforms with unequal lengths cannot be stacked directly.

Padding extends shorter waveforms to the length of the longest example in the batch. It provides a common shape for computation, but the added positions do not belong to the original recording. Lengths and masks preserve that distinction.

Example inputs:

[1, 16000]
[1, 32000]
[1, 24000]

Batch construction:

1. Find the longest clip: T_max = 32000.
2. Zero-pad shorter clips to that length.
3. Stack into [3, 1, 32000].
4. Keep the original lengths and a validity mask.

For each example: mask.sum() = original length. Pooling, attention, and metrics can use the mask to exclude padding.

For example, averaging over every padded time position would include the added region in the denominator. The result would depend on how much padding a clip received. Masked pooling uses only valid positions when aggregating over time. Carrying a mask alongside the waveform makes that information available; each operation that needs it must actually apply it.

The original length also remains useful independently of the padded shape. All three example waveforms occupy 32,000 positions in the batch, while their lengths remain 16,000, 32,000, and 24,000 respectively.

Loading responsibilities:

• Dataset: loads and transforms one example.
• DataLoader: groups examples into batches.
• collate_fn: builds the padded tensor, mask, and lengths.

What the model receives

The classifier receives a batch shaped [B, 1, T]: B recordings, one channel per recording, and a padded time dimension. Convolution layers apply learned filters along time to extract local waveform features. ReLU adds non-linearity between those layers.

The time-dependent features then need to become one representation per recording. Masked mean pooling aggregates valid time positions, and the final linear layer maps the pooled features to K class scores. The output shape is therefore [B, K].

Example classifier in Audio Tensor Lab:

[B, 1, T]
    ↓
Conv1D → ReLU
    ↓
Conv1D → ReLU
    ↓
masked mean pooling
    ↓
Linear
    ↓
[B, K] logits

The output scores are logits. They are not constrained to be between zero and one or to sum to one. Softmax can convert them to probabilities for inspection. During training with CrossEntropyLoss, the raw logits go directly into the loss function, which performs the required log-softmax calculation internally.

• Softmax converts logits to probabilities.
• CrossEntropyLoss takes raw logits and handles log-softmax internally.
• This small classifier demonstrates the data and training path; it is not a speech-recognition model.

From logits to gradients

The forward pass computes predictions using the current model parameters. The loss compares those predictions with the labels and produces a value that training seeks to reduce. Autograd records the differentiable operations that connect that loss to the parameters.

Calling loss.backward() follows the computation graph backward and computes gradients for trainable parameters that contributed to the loss. A gradient describes the local sensitivity of the loss to a parameter: how a small change in that parameter affects the loss near its current value.

Computing gradients does not change the parameters. The optimizer reads the gradients and applies its update rule when optimizer.step() runs. These are two separate stages in the training loop.

batch → forward → logits → loss
                         ↓
                  loss.backward()
                         ↓
                     gradients
                         ↓
                  optimizer.step()
                         ↓
                 updated parameters

optimizer.zero_grad()
logits = model(batch.waveforms, batch.mask)
loss = criterion(logits, batch.labels)
loss.backward()
optimizer.step()

PyTorch accumulates gradients across backward passes by default. In this loop, zero_grad() clears previous gradients so the next update uses the current batch's gradients. Without clearing, earlier gradients would also contribute.

A checkpoint records the state needed later. Model parameters restore the learned computation for inference; resuming training can also require optimizer state, the completed epoch, and the configuration. Saving these together keeps the model and its training state associated with the same point in the run.

Revision points:

• A gradient measures the local sensitivity of the loss to a parameter.
• Backpropagation computes gradients; the optimizer updates parameters.
• Gradients accumulate by default, so the loop clears them before the next backward pass.
• A training checkpoint can store model parameters, optimizer state, completed epoch, and configuration.