openlang

sum(X) on a matrix with one row

A function takes a matrix of observations, rows are observations and columns are features, and normalizes each column with X ./ sum(X). Worked for a long time without trouble. Then an input with one row came in.

sum works along the first dimension whose size isn't 1. For a 1 by n matrix that's the second dimension. So instead of a row of column sums you get one number. No error, no warning.

X = [1 2 3; 4 5 6];
sum(X)       % [5 7 9]
X = [1 2 3];
sum(X)       % 6, not [1 2 3]
sum(X, 1)    % [1 2 3]

The division went through silently, dividing by a scalar is always allowed. Every column should have come out as 1. What came out was the row scaled so that it sums to one. Numbers between zero and one, looked normal.

The fix is an explicit dimension, sum(X, 1). Same for mean and the rest of the functions that pick their dimension from the data. The shape of a result shouldn't depend on how many rows happened to arrive. Fine, I'm writing the dimension into every call now, including the ones that only ever see big matrices.

The conversation continues 6

Follow the reasoning
  1. Permalink

    Fortran fixes rank at declaration. sum(a, dim=1) on a rank-2 array is rank 1 whatever the extents, one row included. The bug here is choosing the dimension from the extents. Pass the dimension every time, in the helpers too.

  2. Permalink

    max has a second output that makes this worse. On a matrix, [m, i] = max(X) gives, for each column, the row where the maximum sits. On a one-row X it gives a single maximum and i becomes a column index. Same variable, different meaning, and if the next line uses i to pick an observation it now picks a feature. [m, i] = max(X, [], 1) keeps i a row index, so for one row it's a vector of ones, one per column.

  3. Permalink

    Julia made the opposite call. sum(X; dims=1) always returns a one-row matrix, even when X has one row, so the shape survives at the price of a vec call when you wanted a plain vector.

  4. Permalink

    So the shape of the result is decided by whatever data the caller sends. Lovely, a function whose return type depends on luck. Fair enough, though, sum(X, 1) is three more characters and the docs say plainly which dimension gets picked.

  5. Permalink

    Right about max, and the same function had a different problem in it. By analogy with sum(X, 1) it said max(X, 1). That's not a dimension. That's the element-wise maximum with 1, so everything below one got pulled up to 1. No error, no warning, same shape. The right call is max(X, [], 1), with the empty placeholder for the second argument. A search through the code for max( and min( with two arguments found three places. Two were real element-wise comparisons and one was the same mistake.

  6. Permalink

    Sounds like one small test would have caught both. An input with one row and values below one, then a check on the shape and the values of the result. max(X, 1) shows up on it right away, and so does sum(X). An input with zero rows could go in too, once you decide what the function should return for it. For a start the one-row case probably covers both of the bugs found so far.

Other sides of the same question

  1. na.rm = TRUE on a group that is all NA

    Per-group totals with tapply(x, g, sum, na.rm = TRUE), and one group came back as 0. Every value in that group is NA. Not a s…

    Replies 4r
  2. A loop that allocated a column at a time

    Slicing in Julia copies, which I knew and then forgot inside a loop. Take a function that walks the columns of a matrix and d…

    Replies 4julia
  3. f2py copies my C-ordered array on every call

    Passing a NumPy array into a Fortran subroutine through f2py. The array is created the usual way, np.zeros((n, m)), so it's C…

    Replies 3fortran