Homework "Artificial Neural Networks"

1. (6 points)

   In this homework we shall construct a simplest artificial neural network which will be trained to interpolate a tabulated function.

   It is an ordinary three-layer neural network with one neuron in the input layer, several neurons in the hidden layer, and one neuron in the output layer,

                              [ hidden neuron ]
                             /                 \w1
                            /                   \
                           /                  w2 \
   x  --->[identity neuron]---[hidden neuron]----[summation neuron]--->  y=Fp(x)
                           \                     /
                            \                   /
                             \                 /w3
                              [ hidden neuron ]
   

   Here the input neuron is an identity neuron: it simply sends the input, a real number x, to all hidden neurons without any modification.

   The output neuron is a summation neuron: it sums the outputs of the hidden neurons and sends the result to the output.

   The hidden neurons are ordinary neurons: the neuron number i transforms its input signal, x, into the its output signal, y_i, as

   yi=f((x-ai)/bi)*wi,
   

   where f is the activation function (the same for all hidden neurons) and where a_i, b_i, w_i are the parameters of the neuron number i.

   The network response F_p(x) is then given as

   Fp(x) = ∑i f((x-ai)/bi)*wi
   

   The activation function can be
   • a Gaussian wavelet, f(x)=x*exp(-x²),
   • a Gaussian, f(x)=exp(-x²),
   • a wavelet, f(x)=cos(5x)*exp(-x²), or any another suitable function.

   The whole network then functions as one big non-linear multi-parameter function y=F_p(x), where p={a_i,b_i,w_i}_i=1..n is the set of parameters of the network.

   Given the tabulated function, {x_k,y_k}_k=1..N, the training of the network consists of tuning its parameters to minimize the cost function

   C(p) = ∑_k=1..N (F_p(x_k) - y_k)²,

   which amounts to minimization of the cost function C(p) in the space of the parameters of the network. This minimization should be done with your own minimization routine.

   A class to keep your network could be like this (put "public" and other access modifiers wherever needed),

   class ann{
   	int n; /* number of hidden neurons */
   	Func<double,double> f; /* activation function */
   	vector p; /* network parameters */
   	ann(int n,Func<double,double> f){/* constructor */}
   	double response(double x){
   		/* return the response of the network to the input signal x */
   		}
   	void train(vector x,vector y){
   		/* train the network to interpolate the given table {x,y}*/
   		}
   }
   

   Train your network to approximate some intersting function, for example

   g(x)=Cos(5*x-1)*Exp(-x*x)
   

   at [-1,1] using, say, the gaussian wavelet as the activation function.

2. (3 points) Modify the previous exercise such that the network, after training, could also approximate the derivative and the anti-derivative of the tabulated function. A gaussian wavelet could be a good activation function here as both its derivative and anti-derivative are analytic.

3. (1 point) Implement an artificial neural network that can be trained to approximate a solution to the differential equation

   Φ[y(x)]≡Φ(y'',y',y,x)=0,

   (where Φ is generally a non-linear function of its arguments) on an interval [a,b] with the boundary condition at a given point 'c',

   y(c)=y_c, y'(c)=y'_c,

   where c∈[a,b] and y_c and y'_c are given numbers.

   The cost function to minimize might be

   δ(p)=∫_a^b|Φ[F_p(x)]|dx +|F_p(c)-y_c|(b-a) +|F_p'(c)-y'_c|(b-a) .

   or

   δ(p)=∫_a^b|Φ[F_p(x)]|²dx +|F_p(c)-y_c|²(b-a) +|F_p'(c)-y'_c|²(b-a) .