Exercises

1.

A time series consists of the numbers 1,0,-1,0,1,0,-1,0, repeated many times. Writing it as a set of points in ${\fam11\tenbbb R}^2$ we get the vectors:

$$% latex2html id marker 7367 \left( \begin{array} {c} 0 \\ 1... ... \left( \begin{array} {c} 0 \\ 1 \end{array} \right), \cdots$$

which we can see is the set of iterations of the matrix

$$% latex2html id marker 7368 \left( \begin{array} {cc} 0 & -1 \\ 1 & 0 \end{array} \right)$$

on an initial vector $\left( \begin{array} {c} 0 \\ 1 \end{array} \right)$. Now suppose some noise is added to this time series, with a zero mean gaussian with variance about 0.1 generating a random number which is added to the original series. This means that iterating from the given (noise free) initial state will produce four clusters in ${\fam11\tenbbb R}^2$. The fact that there are four tells you that the original time series is periodic with period four, and the fact that each cluster is a spherical gaussian tells you that you have additive gaussian noise which is uncorrelated. Filtering the series can be done by replacing each gaussian cluster with its mean.

Experiment by modifying the example so as to (a) increase the period, (b) take longer time delay vectors, (c) change the noise so that it is MA filtered gaussian white noise. Investigate the kinds of point set which you get and decide how you might filter the noise out.

Experiment by taking some matrix map, some initial point, and some noise process. Iterate the map plus the noise and examine the point set you get. If you choose low dimensions you can easily project the sets onto the screen of a computer and gloat over them. What common sense suggestions do you have for filtering out the noise?

2.

Use the fview program (see the bibliography) to obtain samples of the digits zero to nine spoken by, say, five different people. Now get a program to select a sixth speaker, extract the same words and rename the files so as to conceal the words spoken but to store the encoding in case of fights later. Get all your friends to try to work out from looking at the sample trajectories what words the mystery speaker is speaking. Provide a bottle of wine for the person who gets most right. Get the people who got them right to explain how they did it, and try to write a program to use the same methods.

3.

You are given a set of points in the plane in two categories, Noughts and Crosses. The Noughts are represented by the data points:

$$% latex2html id marker 7369 \left(\begin{array} {c}-1 \\ 1 ... ...ght),\left( \begin{array} {c} -0.1 \\ 0.01\end{array} \right)$$

$$% latex2html id marker 7370 \left( \begin{array} {c} 0.0 \\ ... ...ght), \left( \begin{array} {c} 0.9 \\ 0.81 \end{array} \right)$$

and the Crosses by the data points

$$% latex2html id marker 7371 \left( \begin{array} {c} -1 \\ ... ...), \left( \begin{array} {c} -0.1 \\ 0.51 \end{array} \right),$$

$$% latex2html id marker 7372 \left( \begin{array} {c} 0.0 \\ ... ...t), \left( \begin{array} {c} 0.9 \\ 1.31 \end{array} \right)$$

First view the data. This should lead you to suspect that using one gaussian for each category would not be a good idea, and you should also rule out the use of a single unit Perceptron. You then have to choose between the hypotheses that the data shows signs of a group action of transformations which might be learnable, and that it doesn't. In order to test this, you take each data set, a radius of about 0.4, select points and compute for all the points within a radius of this resolution the covariance matrix.

You test to see if the results indicate a dimension of one. (Actually, one is the only one worth trying for in the circumstances, so you could give this a miss.) Having found that this is the case, you decide to try to fit a polynomial function to each category.

Compute a low order polynomial to fit each of the categories. This is your first pass at an attempt at obtaining a transform under which the category is invariant. Proceed as follows: linearise the points of one category by taking the polynomial for the Noughts y = f(x) and sending each point $\left( \begin{array} {c}x\\ y\end{array} \right)$ to the point $\left( \begin{array} {c} x \\ y-f(x)\end{array} \right)$.Now forget about the first co-ordinate and simply store each point by its residue. The results should make pattern recognition rather simpler. Test the legitimacy of using the same polynomial on both data sets by computing the variance of the residues y-f(x) and seeing if the variance is different for each category, or by curve fitting the residue and testing to see how non-linear it is. If the two functions f₀ and f_X for the two categories had been very diffferent, what might you have tried?

4.

Given the simplicity of the last exercise, find an embedding of a k-dimensional cube in ${\fam11\tenbbb R}^n$ for $k \approx 4$ and $n \approx 12$ which gives, when some values are selected randomly and when small amounts of noise are added, a finite data set lying close to a k-manifold This is going to be a data set obtained by applying a k-dimensional space of transformations to a single point. Now change a few parameters in your embedding to get a different embedding of a different data set. Again generate some more points. Note that I took k=1, n= 4; x = t, y=t²; x =t, y=t²+0.5 when I did this in the last exercise. I had zero noise.

Now can you get back again to recover the embeddings given the data points? If you can, describe what happens in the space of embeddings as you go from one category to another. Is there any reason for trying to linearise the space so as to be able to factor out the transformation? After all, it would be simplest to decide what category a datum is in by measuring its distance from the best fitting manifold, why should you go to any more trouble than this? (Hint: There may be more problems in the same space coming down the pipeline!) You might consider the low dimensional case where one category has y = x2 as a fitting manifold and the other has y = x⁴ + x² + 0.3 as its. Find a non-linear transform of ${\fam11\tenbbb R}^2$ which allows you to factor out one component in order to get rid of the transformation group action.

5.

Learning Transformations You are given a set of points of type Nought which lie on the graph of y = x² and points of type Cross which lie along the graph of y = x⁴ + x² + 0.3. All such points have X-values uniformly between -1 and +1. We shall suppose you have plenty of points so as to be able to fit the curves with high precision. You are also given a point of a third category Square, at location $\left( \begin{array} {c} 0.5\\ 0.4 \end{array} \right)$. Where else might you find points of type Square?

6.

The last three exercises had nothing much dynamic about them, and you might feel that they should have been treated earlier with static recognition. Quite so. Now I ask you to produce some pseudo-speech data corresponding to utterances by a range of speakers.

You take a set of measurements of different speakers saying the vowel /AA/, as in English Cat and Bag, and discover, after binning them into 12 filterbank values that the sounds occupy a region of the space ${\fam11\tenbbb R}^{12}$ which is approximated by a single gaussian distribution, and has dimension about six, i.e.

the first six eigenvalues are reasonably large, and the last six are all pretty much the same, small, and can be interpreted as noise. You repeat for seven other vowel sounds and discover that the same is essentially true for each of them, that the centres of the gaussians occupy a plane in the space, and that the gaussians are more or less shifts of each other in this plane. The principal axes of the gaussians all stand out of the plane at an angle of about 30^o, like six dimensional whales leaping out of a flat sea, all about half out and all pointing in the same direction. An impressive sight.

You entertain the hypothesis that the distributions are telling you that there is a six dimensional space of transformations which can be performed on a vowel utterance in order to encode information about how loudly it is spoken, the age and sex of the speaker, and so on.

(a)

How might you test this hypothesis?

(b)

If you were satisfied that the hypothesis is tenable, how would you use the data in order to normalise a new trajectory in the vowel space?

(c)

What about a trajectory representing a word such as `` wish'' or ``whiff''?

(d)

Construct some data satisfying the conditions described and examine it carefully, comparing it with real speech data.

7.

Consider the double spiral data set of the last chapter. Although somewhat implausible as a model for some transforms of a basic category, it is plain that there is a certain similarity between the case of an embedding of an interval arising from some family of transformations and the spiral structure of the data sets. In fact one could represent the spirals as the embedding of an interval. The question arises, can one extend the arguments used to deal with embeddings of manifolds which are not too far from being flat, to cases such as the double spiral in which there is plainly some kind of structure to the data? It is plain that we could perform the same trick as before: first choose a suitable radius, then compute covariance matrices for points lying inside balls of the radius, use the coincidence of all of them having one eigenvalue much bigger than the other to argue that the data is essentially one dimensional, and then try to fit a curve. It is the last part which will give some troubles to an automatic system; doing it with polynomials does not usually give good extrapolation

properties. Try it and see.