Something that's reasonable to expect to see in Lua files is something
like:
local sel4 = sel1 - sel2 - sel3
or more generally, producing a selection from subtraction that will then
be used in subsequent selection math.
I discovered this wasn't actually working correctly, and that it also
applied to the xor operation. The reason behind this is that
l_selection_sub and l_selection_xor create a new selection from nothing,
which by default has "lower" bounds of COLNO, ROWNO and "upper" bounds
of 0,0. Iterating across the intersecting rectangle of both selections
does not reliably set the bounds of the resulting selection properly,
since the first selection_setpoint with a value of 0 will cause the
selection's bounds_dirty flag to be set, at which point they will cease
to change as more points are added.
Then this selection with its incorrect boundaries is pushed back onto
the Lua stack, and becomes the first operand of the next subtraction
(i.e. selr in the first l_selection_sub becomes sela in the second
l_selection_sub). Depending on how broken the bounding box is, results
may vary, but if the bounding box is still (COLNO,ROWNO,0,0), the
resulting selection will have no points selected at all.
This fixes this problem by forcibly recalculating the bounds of the
result selection, so any subsequent operations on it will be valid.
c6fa9c3099