{"id":22746,"date":"2021-04-16T07:46:00","date_gmt":"2021-04-16T07:46:00","guid":{"rendered":"https:\/\/www.experfy.com\/blog\/neural-network-anatomical-structures\/"},"modified":"2023-08-26T06:52:48","modified_gmt":"2023-08-26T06:52:48","slug":"neural-network-anatomical-structures","status":"publish","type":"post","link":"https:\/\/www.experfy.com\/blog\/ai-ml\/neural-network-anatomical-structures\/","title":{"rendered":"Two Fundamental Neural Network Anatomical Structures"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Fully-connected versus Convolutional<\/p>\n

Like other organisms, artificial neural networks have evolved through the ages. In this post, we cover two key anatomies that have emerged: fully-connected versus convolutional. The second one is better suited to problems in image processing in which there are local features in a space with geometry. The first one is generally appropriate on problems in which there isn\u2019t a geometry and spatial locality of features is not paramount.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Single Neurons<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Let\u2019s start with models of single artificial neurons, the \u201cLeggo bricks\u201d of neural networks. A neuron takes a vector x<\/strong> as input and derives a scalar output y from it. Most neuron models conform to y = f(wx + <\/strong>b). Here w <\/strong>is the vector of weights of the same dimensionality as x<\/strong>, and b is a scalar called the neuron\u2019s bias. This is graphically depicted below.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Two\t\t\t\t\t\t\t\t\t\t\t
Multiply the input vector with a weight vector, then feed through an activation function<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The configurable part of this is the neuron\u2019s activation function f. Different choices of f<\/em> lead to neurons with quite different capabilities.<\/p>\n

The most common choices are linear: f<\/em>(u<\/em>) = u<\/em>, step: f<\/em>(u<\/em>) = sign(u<\/em>), and sigmoid: f(u) = 1<\/em>\/(1+e^-u<\/em>). These are depicted below.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Activation\t\t\t\t\t\t\t\t\t\t\t
Activation functions: sigmoid, linear, step<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

When the target output y is expected to be (approximately) a linear function of x<\/strong>, a linear activation function is called for. This is the setting of linear regression.<\/p>\n

When the target output y is binary, we have a choice: a sigmoidal activation function or a step activation function. Both theory and practice favor the sigmoid.<\/p>\n

There are several reasons for this.<\/p>\n

One is that the sigmoid is differentiable whereas the step function is not. This allows the neuron\u2019s parameters (the weights\u00a0w<\/strong>\u00a0and the bias b) to be trained via gradient descent on a data<\/a> set of input-output pairs.<\/p>\n

A second one is that using a sigmoid captures more information which can often be used profitably. For example, consider two situations, one in which the neuron\u2019s output is 0.9 and one in which it is 0.7. We might be inclined to classify both as 1s. (Remember we seek a binary output.) Should we do so, it makes sense to attach higher confidence to the first one since the neuron\u2019s output was higher.<\/p>\n

Another way we can use the additional information in the neuron\u2019s output is by adjusting the binary classification threshold. This lets us become more (or less) conservative in our decision-making.<\/p>\n

The key point here is that the classification threshold can be adjusted post-training. In fact, any time we wish to. (This is especially useful after the neuron starts making decisions in the \u201cfield\u201d and we realize we\u2019d like to tweak its behavior.) This threshold is not a parameter during training. Only the weight vector w <\/strong>and bias b are.<\/p>\n

Simply put, if we observe the neuron is overly sensitive (i.e. its outputs are sometimes towards 1 when the target is 0) we can increase the classification threshold. Similarly, if the neuron is not sensitive enough, we can decrease the classification threshold.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Networks of Neurons<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Well, actually a single neuron is already an example. A useful one at that. It can be used to map a vector of inputs to a numeric or binary output. That is, to solve regression and binary classification problems.<\/p>\n

Often, not very well though.<\/p>\n

The first breakthrough is an intermediate layer of neurons between the input and the output. This is graphically depicted below.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Two\t\t\t\t\t\t\t\t\t\t\t
A hidden layer between X and y. Sigmoidal hidden neurons are especially attractive<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The intermediate layer is called a hidden layer. In the schematic, we have used sigmoidal neurons in the internal layer. Linear hidden neurons are less useful and step neurons have issues we discussed earlier.<\/p>\n

A sigmoidal hidden neuron may be viewed as representing some binary feature of the input. The neuron\u2019s value is derived from the input vector x<\/strong>. This value may be viewed as the probability that the associated feature is present in the input.<\/p>\n

A neural network with a hidden layer maps an input vector to a vector in a space of features. This mapping is nonlinear. The feature vector is then mapped to the output. This indirect approach results in an architecture that is in principle more powerful than one without the hidden layer.<\/p>\n

In practice, there are some issues. How many neurons should go into the hidden layer? This depends on the complexity of the input-to-output mapping problem. This complexity may not be known. For linear problems, we may not need any hidden neurons. In fact, having them might hurt.<\/p>\n

The short answer is we don\u2019t. That said, we could try a different number of hidden neurons and pick the one that works best or at least adequately.<\/p>\n

On to the next question. For a fixed number of hidden neurons, how do the features they represent get learned? The short answer is via the learning process, typically a form of gradient descent called back-propagation. We won\u2019t go into the details here.<\/p>\n

That said, we will depict the roles the various weights touching a hidden neuron play.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Feature\t\t\t\t\t\t\t\t\t\t\t
The input-to-feature weights learn the feature function. The weights from the various features to the output control the relative influence of the various features on the final output.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Finally, let\u2019s mention that, as before, the output neuron is sigmoidal for a binary classification problem and linear for a regression problem.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Image Classification Problems<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Say we have a large set of images, some containing cats, others not. We\u2019d like to learn a classifier that can tell whether or not an image has a cat in it.<\/p>\n

Say each of our images has lots of pixels in them. 100 X 100 = 10,000 for concreteness.<\/p>\n

In principle, we could map this problem to a neural network with one hidden layer, as depicted below.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\"Two\t\t\t\t\t\t\t\t\t\t\t
Lots of input neurons. With even a moderate number of hidden neurons, this means lots and lots of input-to-hidden weights!<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In practice this approach has difficulties. We have 10,000 input neurons. With a hidden layer of m<\/em> neurons, this means we have 10000*m<\/em> input-to-hidden weights. It’s hard to imagine that we could use less than 20 hidden neurons to adequately learn a cat-or-not classifier.<\/p>\n

20 hidden neurons means 200,000 input-to-hidden weights. That’s a lot of weights to learn! Even with a rich training set, overfitting is a significant risk.<\/p>\n

Let\u2019s think differently. Is there structure in this domain (images) that we might be able to exploit? It turns out the answer is yes. First some observations.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
  1. The input neurons are in a spatial grid.<\/li>
  2. Features in an image are often local.<\/li>
  3. The same feature may occur at different locations in the input.<\/li><\/ol>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Let\u2019s expand on 2 and 3. Consider the picture below. It shows horizontal edges at various locations. Each edge is the same feature but at a different location. Each of these feature occurrences is also local. Local just means that to detect the edge at any one location, one only needs to look at pixel values in the proximity of the edge.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\"Local\t\t\t\t\t\t\t\t\t\t\t
    A local feature (horizontal edge) that occurs at many locations in the image<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Okay, we now see that each feature is also on the same spatial grid as the input pixels. The implication of this is that for any one feature, there isn\u2019t a single value, rather a grid of values. For each location (i,j), feature f<\/em> has a value that indicates whether f is present or absent at that location.<\/p>\n

    Great. Seems like we have gone in the opposite direction. Seems like we have made the problem more complex. Instead of a feature having a single value, it now has a spatial grid of values, one per location.<\/p>\n

    Not really. How would we try to represent a local feature in an MLNN? Separating out the actual feature function from the location where it applies? Plus, leverage its locality. We can\u2019t.<\/p>\n

    The inability of the MLNN to (i) exploit feature locality and (ii) the ability to evaluate the feature at many different locations means that we need a lot of hidden neurons to model the combination of the functional and spatial aspects of a feature. (Functional meaning what the feature is, spatial meaning where it is evaluated.) On top of that, because we are unable to exploit locality, the number of input-to-hidden weights explodes.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    lots of input neurons X lots of hidden neurons X fully-connected \u2192 lots and lots of input-to-hidden weights \u2192 Network way overly complex<\/pre>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    The picture below depicts the alternative that leverages the input geometry, the locality of features, and the need to evaluate a feature at many different locations.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\"Two\t\t\t\t\t\t\t\t\t\t\t
    The value of a feature is itself a map. The feature\u2019s function is scanned over the entire input image to produce its feature map.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    Let\u2019s zoom into the feature\u2019s neuron at location (i,j).<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\"Locality\t\t\t\t\t\t\t\t\t\t\t
    Locality made explicit with the receptive field. The feature\u2019s weights depend only on its function, not on the location it is evaluated.<\/figcaption>\n\t\t\t\t\t\t\t\t\t\t<\/figure>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    Okay, so any single (local) feature is represented by the same small set of weights. We just slide this feature\u2019s function (called a Kernel) over the various locations (i<\/em>, j<\/em>) to get a reading on the feature\u2019s values over the entire grid. This sliding process is called convolution.<\/p>\n
    By contrast, the MLP has no explicit mechanisms for exploiting either locality or sharing of weights.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t
    Summary<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    We have covered the anatomical structures of the two most important (feedforward) neural network architectures: fully-connected multi-layer neural networks and convolutional neural networks. We have discussed why convolutional neural networks are better suited to image processing than multi-layer neural networks. On the MLNNs are well-suited to problems in which locality and convolutions don\u2019t come into play in obvious ways.<\/p>\n
    Further Reading<\/strong><\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    1. http:\/\/deeplearning.stanford.edu\/tutorial\/supervised\/ConvolutionalNeuralNetwork\/<\/a><\/li>
    2. http:\/\/deeplearning.stanford.edu\/tutorial\/supervised\/MultiLayerNeuralNetworks\/<\/a><\/li><\/ol>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
      Artificial neural networks have evolved through the ages. In this post, we cover two key anatomies that have emerged: fully-connected versus convolutional.<\/p>\n","protected":false},"author":1044,"featured_media":19167,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[183],"tags":[97,297,1500,1501,1502],"ppma_author":[3691],"class_list":["post-22746","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-ml","tag-artificial-intelligence","tag-artificial-neural-networks","tag-convolutional-network","tag-multilayer-perceptron","tag-sigmoid-function"],"authors":[{"term_id":3691,"user_id":1044,"is_guest":0,"slug":"arun-jagota","display_name":"Arun Jagota","avatar_url":"https:\/\/www.experfy.com\/blog\/wp-content\/uploads\/2021\/05\/Arun-Jagota-150x150.jpeg","user_url":"https:\/\/www.salesforce.com\/in\/?ir=1","last_name":"Jagota","first_name":"Arun","job_title":"","description":"Arun Jagota is Director of Data Science at Salesforce.com. A PhD in computer science, he has taught undergraduate, graduate, and continuing education courses in Computer Science at many US Universities from 1992 through 2006. He has written a number of books, most available at Amazon.com, 50 academic publications and has 17+ patents issued."}],"_links":{"self":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22746","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/users\/1044"}],"replies":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/comments?post=22746"}],"version-history":[{"count":4,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22746\/revisions"}],"predecessor-version":[{"id":31559,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/posts\/22746\/revisions\/31559"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media\/19167"}],"wp:attachment":[{"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/media?parent=22746"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/categories?post=22746"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/tags?post=22746"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.experfy.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=22746"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}