The computer that von Neumann built at the Institute for Advanced Study (IAS) was not the first computer, but it was probably the most influential. Turing's Cathedral is its story and of much else.
To get a feel for computing circa 1945, let's start by relating the well known fact that 'computer' at this time meant a person (usually a woman) performing calculations with the aid of mechanical desk calculators. The proposal for ENIAC (a well known predecessor) coined the term 'computor' to refer to a machine (which seems not to have caught on though). The architecture of ENIAC with its 20 accumulators was in fact a replication of the setup where 20 human computers were performing their part of a calculation and passing results back and forth; effectively a distributed algorithm migrated from human nodes to electronic ones (Incidentally, with humans storage was cheap but multiplication expensive. With ENIAC the situation was reversed, and all the algorithms had to be revised.) The architecture that has become familiar since, with a CPU, hierarchical memory etc, was what the IAS computer pioneered. Moving to hardware, consider the memory unit. The IAS computer had 1024 forty bit words of memory, a fortune at the time. The basis for implementation was an oscilloscope tube, which was on its way to evolve as CRT display. Wide surface of the tube was divided in a 32 by 32 grid, each referring to a binary digit, accessible via appropriate deflections of the electron beam for both reading and writing, with a 24us access time. Forty such tubes operating in parallel, and with corresponding locations in each together forming a single (40 bit) word was how the memory functioned. The machine was built out of vacuum tubes, since the transistor was not yet invented and binary digits were referred to as bd's, as the word ‘bit’ was not yet invented either. It had an instruction set (unlike the ENIAC where (at least initially) the instruction set was implemented as plugboard settings), and each word would contain an opcode, as well as an address (the famous von Neumann architecture). On the software side, in its ~5KB memory, the machine was used to run climate simulation models, simulations of gene-like replicators, and of course lots of secret weapons calculations. von Neumann was quick to see the possibilities of a universal machine (once Turing had laid the groundwork) and went about building his machine as a scientific instrument, away from both industry and government. He actively put all the information in public domain, allowing a number of replicas to emerge and the ideas to be improved upon.
The book is ‘intensely human’ (as one reviewer put it) at its core and introduces a remarkable cast of characters. There is von Neumann of course, who made significant contributions to mathematics, physics, economics, computing, and much else. Less well known is his wife, Klara, one of the original programmers of the machine. Julian Bigelow was the main engineer, the person most responsible for making the machine a reality. Stanislaw Ulam was von Neumann's friend, and maker of a number of original contributions to mathematics and physics, among them the Monte Carlo method. Leibniz and Gödel, Turing and Teller, all make an appearance, taking their place in the ever widening stream of ideas and events.
For a long time, I wanted to read a biography of von Neumann, and while this book is not strictly one, it fits the bill nicely. It can be classified as a history of a particular computer, but that would be very limiting. This energetic romp through the thicket of philosophical, mathematical and physical ideas that came together in the form of the computer was a source of genuine exhilaration. It may be so for you too.