Neural Networks
A neural network is a web of simple units that each do tiny arithmetic: multiply inputs by learned weights, add them up, and pass the result on. Intelligence emerges from millions of these units working together.
Learning Objectives
- Describe what an artificial neuron computes, in plain arithmetic.
- Explain how neurons connect into layers and networks.
- Trace one number through a tiny network by hand.
- Explain weights as "the knobs training adjusts."
- Connect network size (parameters) to model capability and cost.
Why This Matters
Neural networks are the machinery inside everything from face unlock to ChatGPT. The surprise is how simple the parts are: no unit "thinks"; each does arithmetic a ten-year-old could check. Understanding this demystifies AI and makes sense of terms you will meet constantly: parameters, weights, layers, and why "175 billion parameters" describes a model's size.
Everyday Analogy
Picture a huge orchestra where each musician plays one note, listening only to a few neighbors. No musician knows the symphony. Yet with the right sheet music, specifying who plays when and how loudly, the ensemble produces something no individual could. In a neural network, each neuron is a musician playing arithmetic; the weights are the sheet music; and training (AI-005) is the process of writing that music from examples.
The Neuron: Simpler Than You Think
One artificial neuron does exactly three things:
- Multiply each incoming number by a weight (its personal importance dial).
- Add the results together, plus a bias (a baseline offset).
- Squash the total through a simple function (e.g., "if negative, output 0") and pass it on.
That's the whole unit. Written out for a neuron with three inputs:
output = squash( x1×w1 + x2×w2 + x3×w3 + bias )
The intelligence is not in the neuron. It is in the values of the weights, which training sets.
Worked Example: A Two-Neuron Decision
A tiny network deciding "should I take an umbrella?" from two inputs: cloud cover (0–1) and humidity (0–1).
Neuron A (rain detector): weights [cloud: 0.8, humidity: 0.6], bias −0.5 Today: cloud = 0.9, humidity = 0.7 → 0.9×0.8 + 0.7×0.6 − 0.5 = 0.64 → positive → outputs 0.64
Neuron B (umbrella decision): weight [rain-signal: 1.2], bias −0.5 → 0.64×1.2 − 0.5 = 0.27 → positive → take the umbrella.
Follow the arithmetic; there is nothing else hidden. Now imagine not 2 neurons but 100 billion, arranged in a hundred layers, with weights set not by hand but learned from trillions of examples. Same math. That is GPT-4-class machinery.
From Neurons to Layers to Networks
Neurons stack into layers; each layer's outputs feed the next layer's inputs:
- Input layer: the raw numbers (pixel values, token IDs).
- Hidden layers: the pattern-building middle (edges → parts → objects, as in AI-003).
- Output layer: the answer (probabilities per category, or the next word).
Parameters = all the weights and biases in the network. This is the number in every model card:
| Model | Parameters |
|---|---|
| Tiny image classifier | ~100 thousand |
| Face recognition network | ~25 million |
| GPT-2 (2019) | 1.5 billion |
| Frontier LLMs | hundreds of billions (see AI-064) |
More parameters = more pattern-storage capacity = more data and compute needed to train and run (a trade explored in AI-065).
Where the Knowledge Lives
A crucial mental model: after training, everything the network "knows" is encoded in its weight values. There is no database inside, no stored sentences, no lookup table. Just billions of numbers whose collective arrangement transforms inputs into remarkably good outputs. This is why models can't cite where a fact came from, and why they sometimes confabulate (AI-060): knowledge is dissolved into arithmetic, not filed in folders.
Real-World Showcase
- Your phone's face unlock runs a ~10M-parameter network dozens of times per second on a chip smaller than a fingernail.
- AlphaGo's networks evaluated board positions no human had articulated rules for; the weights encoded intuition-like judgment learned from millions of games.
- Hearing aids and noise-cancelling earbuds now run tiny neural networks in real time to separate speech from noise.
Try It Yourself
- Compute a neuron by hand: inputs [0.5, 0.8], weights [1.0, −0.5], bias 0.2. (Answer: 0.5×1.0 + 0.8×(−0.5) + 0.2 = 0.3 → positive → output 0.3.) You just executed the core operation of every AI model.
- Redo the umbrella example with a sunny day (cloud = 0.1, humidity = 0.3) and confirm the network says no umbrella. Then change one weight and watch the decision change. You just experienced why weights are everything.
- Search "TensorFlow Playground" in a browser: drag neurons and layers, press play, and watch a network learn a spiral pattern live.
Common Mistakes
- Imagining neurons as brain cells: the analogy inspired the name, but these are arithmetic units, vastly simpler than biological neurons.
- Thinking some neuron "contains" a concept, when concepts are spread across millions of weights collectively.
- Equating more parameters with more intelligence, when capacity helps only with enough quality data to fill it (AI-006).
- Believing the network stores its training data, when it stores only the patterns, compressed into weights.
Key Takeaways
- A neuron = multiply by weights, add, squash. Nothing more.
- Networks = neurons in layers; outputs of one layer feed the next.
- Weights and biases (= parameters) are where all learned knowledge lives.
- "175B parameters" means 175 billion learned numbers: capacity, not wisdom.
- The same arithmetic scales from your umbrella example to frontier LLMs.
Glossary
- Neuron
- A single unit that does three things: multiply each input by a weight, add the results plus a bias, and squash the total through a simple function before passing it on. The umbrella example shows the full arithmetic; there is nothing else hidden inside.
- Weight
- A learned number acting as a neuron's importance dial for one input, like 0.8 for cloud cover in the rain detector. In the orchestra analogy the weights are the sheet music; the intelligence lives in their values, not in the neurons.
- Bias
- A baseline offset added to a neuron's weighted sum before squashing, such as the −0.5 in the umbrella example. It shifts the threshold at which the neuron activates.
- Activation Function
- The "squash" step that transforms a neuron's summed input before passing it on, e.g. "if negative, output 0." It is what lets stacked layers build patterns more complex than plain addition and multiplication.
- Layer
- A group of neurons whose outputs feed the next group: input layer (raw numbers), hidden layers (pattern-building middle), output layer (the answer). Layer stacking is what makes networks deep. (see AI-003)
- Parameters
- All the weights and biases in a network. It's the number in every model card, from ~100 thousand in a tiny classifier to hundreds of billions in frontier LLMs. More parameters means more pattern-storage capacity, and more data and compute to train and run. (see AI-064)
- Hidden Layer
- Any layer between input and output where intermediate patterns are built: edges into parts into objects. The name just means its values are not directly observed as input or output.
- Confabulation
- Producing plausible but wrong output because knowledge is dissolved into weight arithmetic rather than filed in a lookup table. This is why a network cannot cite where a fact came from. (see AI-060)
References
Diagram
Knowledge Check
8 questions