Homework "Artificial Neural Networks"

1. (6 points)

   In this exercise we construct a simplest artificial neural network which will be trained to interpolate a tabulated function.

   It is an ordinary three-layer neural network with one neuron in the input layer, several neurons in the hidden layer, and one neuron in the output layer,

                              [ hidden neuron ]
                             /                 \w1
                            /                   \
                           /                  w2 \
   x  --->[identity neuron]---[hidden neuron]----[summation neuron]--->  y=Fp(x)
                           \                     /
                            \                   /
                             \                 /w3
                              [ hidden neuron ]
   

   Here the input neuron is an identity neuron: it simply sends the input, a real number x, to all hidden neurons without any modification.

   The output neuron is a summation neuron: it sums the outputs of the hidden neurons and sends the result to the output.

   The hidden neurons are ordinary neurons: the neuron number i transforms its input signal, x, into the its output signal, y_i, as

   yi=f((x-ai)/bi)*wi,
   

   where f is the activation function (the same for all hidden neurons) and where a_i, b_i, w_i are the parameters of the neuron number i.

   The network response F_p(x) is then given as

   Fp(x)=∑i f((x-ai)/bi)*wi
   

   The activation function can be

   • a Gaussian wavelet, f(x)=x×exp(-x²),
   • a Gaussian, f(x)=exp(-x²),
   • a wavelet, f(x)=cos(5x)×exp(-x²),
   or any another suitable function.

   The whole network then functions as one big non-linear multi-parameter function y=F_p(x), where p={a_i,b_i,w_i}_i=1..n is the set of parameters of the network.

   Given the tabulated function, {x_k,y_k}_k=1..N, the training of the network consists of tuning its parameters to minimize the cost function

   C(p)=∑_k=1..N(F_p(x_k)-y_k)²,

   which amounts to minimization of the cost function C(p) in the space of the parameters of the network. This minimization should be done with your own minimization routine.

   Your ann.h header file could be like this,

   #include"gsl/gsl_vector.h"
   #ifndef HAVE_ANN_H
   #define HAVE_ANN_H
   typedef struct { int n; double(*f)(double); gsl_vector* params; } ann;
   ann*   ann_alloc   (int n,double(*f)(double));
   void   ann_free    (ann* network);
   double ann_response(ann* network,double x);
   void   ann_train   (ann* network,gsl_vector* xs,gsl_vector* ys);
   #endif
   

2. (3 points) Modify the previous exercise such that the network, after training, could also approximate the derivative and the anti-derivative of the tabulated function.

3. (1 point) Implement an artificial neural network that can be trained to approximate a solution to the differential equation

   Φ[y(x)]≡Φ(y'',y',y,x)=0,

   (where Φ is generally a non-linear function of its arguments) on an interval [a,b] with the boundary condition at a given point 'c',

   y(c)=y_c, y'(c)=y'_c,

   where c∈[a,b] and y_c and y'_c are given numbers.

   The cost function to minimize might be

   δ(p)=∫_a^b|Φ[F_p(x)]|dx +|F_p(c)-y_c|(b-a) +|F_p'(c)-y'_c|(b-a) .

   or

   δ(p)=∫_a^b|Φ[F_p(x)]|²dx +|F_p(c)-y_c|²(b-a) +|F_p'(c)-y'_c|²(b-a) .