Book Review: Hamming's Learning to Learn

There are a certain percentage of people who are able to master the art of learning itself. You can think of people like Tim Ferriss or Naval Ravikant who have figured out just how to get to the right level of proficiency that would enable them to then compound their learning exponentially if they decided to do so. There are also the archetypes who just have brilliance baked in; too many to name but my friend Dirac would be such person. The learning to learn feature is a skill that is hard to teach but is more about observation and practice. Richard W. Hamming is one such person who figured out the art of learning and leveraging it to solve serious problems and on top of that was able to write and communicate it effortlessly.

The book titled "The Art of Doing Science and Engineering: Learning to Learn" by Richard W. Hamming is along these lines [1]. It's a read on the accounts and recollections of a master at learning and leveraging what has been learned to solve problems through thought and questioning. What you find in Hamming's book is that he continuously leverages the tools, ideas, and nuances that he was familiarized with throughout his career. By training he was a mathematician but through the need to solve problems became a computer scientist. And through the advancements and burgeoning field of computers he became a digital signal processing pioneer.

The Bell Lab Phenomenon

Bell Labs was one of the leading research and technology companies in the world from the 1940s to the 1980s. It was here that luminaries such as Hamming, Shannon, Shockley, Shor, and many others made some of the most significant discoveries in human history. It was arguably one of the most prolific institutions in the world for scientists and engineers.

The book is a collection of topics Hamming worked on throughout his career being told from the perspective of a teacher to his students. Throughout it the chapters of some quintessential passages that are bolden to highlight the importance. My favorite quote in the book is:

Luck favors the prepared mind. – Paraphrased from Louis Pasteur

I think this rings so true in two ways. The first is that "Luck" is just that, things that happen by chance due to the fact that you were unaware of all possible outcomes or information. There is nothing you can do to be lucky, in the purest meaning of the word, its just all about what comes across your path. The second is that with time, mental clarity, and skill preparation, when opportunities present themselves, you are able to take advantage of them where as the amateur would have let it go unnoticed.

Here are some other great passages — commonly repeated by other greats in science and engineering — from the book that are pertinent to the current AI/ML wave:

"Learning a new subject is something you will have to do many times in your career if you are to be a leader and not left behind as a follower by new developments." p. 180

"Those who did make significant contributions viewed computers as something new, studied on their own merits ... people always want to think something new is just like the past ... and hence they prevent themselves from making any significant contribution to the new field being created under their noses." p. 196

"to the extent you can choose, work on problems you think will be important." p. 228

"Why should anyone believe the simulation is relevant?" p. 246

I would recommend this book to about anyone who is looking for a leisure read about how Hamming learned to learn and think. Its probably best for someone who is a senior in in college or a first year grad student as it will prime their mind to be alert for how to improve themselves. There are at times when the specific math takes some thinking through (e.g. error correcting codes) but nothing is that daunting to read through and comprehend. Hamming does state in the beginning not to worry about the math but just follow along with his thinking at the time for how to address the problems. The style of writing is very engaging and never a dull moment, especially because Hamming did so much at Los Alamos and Bell Labs.

The one thing this book is not is a tutorial or hands-on book for learning to learn. Its not about how to prepare your notes, listen to lectures, use flashcards, or any other mnemonic device. It is all about how to ask yourself questions that probe thinking and applying your learned experiences to answer those questions and thoughts.

---

References

[1] R. Hamming, The Art of Doing Science and Engineering: Learning to Learn. N.p., Stripe Matter Incorporated, 2020. Google

---

Reuse and Attribution

Sept: Reading & Thinking

I don't have too much to write about these days as I'm spending a lot of my free time reading various books. I've been doing some reading on superconducting materials given all the attention in July and August on LK-99. Most the texts I'm reading on this topic are introductory so they don't cover the theory that well, which I would eventually like to have a good understanding of.

I've also been focusing on some computational neuroscience via the online course. Its pretty interesting how underlying model for neurons is some variation of an RC circuit. This is captured with the nobel prize winning model proposed by the Hodgkin-Huxley model, which is a gated version of a RC circuit and described by a set of nonlinear ordinary differential equations¹. The cool thing is its very easily to play around with this type of model using the SciML Julia language framework, specifically the DifferentialEquation.jl package. I'm in the process of creating a computational blog entry on it.

Finally, I've been thinking about some self-driving lab activity using a 3D printer. I would love to use a metal/ceramic FDM²/FFF³ than a polymer-based printer. It looks like there are some good options by Markforged and Desktop Metal. I don't know much about the technology used by these manufactures or the feasibility of doing materials development. What my dream would be is to tie a 3D printer like those into some filament synthesis and printed part testing apparatus. Then using concepts of self-driving labs, we can define a goal/policy for the system to work towards. For example find the pareto front⁴ for optimal tensile strength and minimal density. I've contributed to a awesome-self-driving-labs github repo that has some good resource materials to learn about this field.

Footnotes

---

1. A nonlinear ordinary differential equation (ODE) is an equation involving a function and its derivatives in which the relationship between the variables is not proportional. In mathematical terms, if $F(x, y, y')$ represents the ODE, it is nonlinear if it can't be written as a linear combination of $y$ and its derivatives. These equations frequently emerge in complex systems such as biological processes. For example, the equation $\frac{dy}{dx} = y^2$ is nonlinear because of the $y^2$ term. Unlike linear ODEs, a small change in initial conditions can lead to substantial differences in outcomes, making them inherently more complex to solve and analyze. ↩

2. Stands for fused deposition modeling. Its a poor name if you ask me but I believe the name comes from the fact that STL files are used to print single printed object. ↩

3. Stands for fused filament fabrication. Almost the same as FDM with some minor differences. ↩

4. The pareto front or frontier is the set of solutions that provide a the best options between a multi-objective function. It captures the trade-offs. ↩

Reuse and Attribution

Materials Informatics Book

Should I write a technical "how-to" textbook?

Why am I asking this question, well, I've come to somewhat of a cross roads in materials informatics. I have digested a lot of research papers and review monographs [1-4], but constantly run into the following problem:

How do I implement the technique discussed, reproduce the results, and extend it towards my specific domain topic?

The answer over and over is, these resources aren't going to enable that or provide a way to do so. Yes in some cases there are Github repos that provide the source, but what I find is the data input pipeline is so complicated or convoluted that figure out how to get my data to work is too much effort. It would just be better to actually implement the model from scratch based on my specific data preprocessing/pipeline.

So why would me writing a address this challenge? Well, for one no one else has written such a book. You have three textbooks available on this topic:

• Isayev, O., Tropsha, A., & Curtarolo, S. (2019). Materials informatics: Methods, tools, and applications. John Wiley & Sons. url.
• Rajan, K. (2013). Informatics for materials science and engineering: Data-driven discovery for accelerated experimentation and application. Butterworth-Heinemann. url.
• Kalidindi, S. R. (2015). Hierarchical materials informatics: Novel analytics for materials data. Elsevier. url.

There are a few other monographs [3-4] that try to focus on specific subdomain areas of materials science. The books above are actually pretty good if your looking for a foundational understanding of data science and machine learning applied to materials science and engineering. The problem with those books is they are more reference text for people involved in materials informatics. They won't help you get going in front of a computer or information systems.

This is what is missing and I want to provide a solution. One reason is that I'll learn more by writing a book. The second reason is I think the tools are now available to make writing this type of book much smoother. Writing literate programming is a regular thing (e.g., Jupyter notebooks) and therefore writing while coding is straightforward, usually.

The dilemmas I face are what programming language to use, what framework to write in, how much to cover, and best examples/case studies to use. For the programming language its between Python and Julia. I'm torn because I prefer to use Julia but Python is more broadly adapted and has very mature and standard packages (e.g. scikit-learn, pytorch). For the framework its also a challenge. I'm favoring Quarto at the moment and it won't matter if I use Julia or Python. Similar case for Jupyter Books. I haven't used Jupyter Books and I'm not too interested in adapting the MyST. There are other options to explore as well, such as Books.jl which is geared towards PDF and website generation.

For the content the book would cover, I need to be very thoughtful. My rough outline would be something like:

1. What is data and information
2. Describing data: Probability, Statistics, and Visualization
3. Processing & Transformation of data
4. Pattern extraction and reduced representation
5. Regression, optimization, and prediction
6. Neural network models
7. Autonomous solution seeking

The first 3 chapters are probably self-explanatory, the chapters 4,5, and 7 would correspond to unsupervised, supervised, and reinforcement learning. This is a moving target so it would change based on my particular interest and focus.

The key point is that each chapter provides the background and the code to actually do something at the computer. My goal would also be to try to do as much as possible from scratch. Meaning, that if it made sense I would actually have a section in the chapter on neural networks that builds the layers, does forward and backward propagation, and trains using minimal packages (e.g. NumPy). Why do so if in the end we all are going to implement and deploy using pytorch or `tensorflow? Because for most the act of doing is what solidifies understanding and comprehension. After coding up a simple NN, when someone talks about backpropagation, you'll know what is actually being down, at least from the most minimal implementation ¹.

Status

I have yet to really start writing, but plan to have some kind of a draft by end of 2024. My goal would be to make the draft available online first. For the physical copy I intend to go the self-publishing route using Amazon Kindle services.

As for the applications and case studies used, well, I really want these to be real in the sense that they have either been done in academia or industry. I want to try and avoid "toy problems" not because they aren't useful but because I want to avoid the issue with a creating a insurmountable barrier to applying whats in the book the readers specific interest/problem.

My hope for this potential book is that grad. students and researchers who want to get into this area, but don't have any hands-on experience, can more easily do so by working through the book. The book obviously wouldn't be at the forefront of research methods, but it would be as if you were taking a graduate level lab at a major university.

References

[1] K. Takahashi and L. Takahashi, "Toward the Golden Age of Materials Informatics: Perspective and Opportunities", J. Phys. Chem. Lett., vol. 14, no. 20, pp. 4726-4733, May 2023, doi: https://doi.org/10.1021/acs.jpclett.3c00648.

[2] C. Li and K. Zheng, "Methods, progresses, and opportunities of materials informatics", InfoMat, p. e12425, Jun. 2023, doi: https://doi.org/10.1002/inf2.12425.

[3] T. Lookman, F. J. Alexander, and K. Rajan, Information science for materials discovery and design. Springer, 2015.

[4] I. Tanaka, Nanoinformatics. Springer, 2018.

Footnotes

---

1. There are different numerical implementations to achieve backpropagation and the book I would write showing how to implement a NN would focus on the most basic approach. ↩

Reuse and Attribution

Book Review: Quantum Entanglement, Jed Brody

I just finished reading through this short monograph on quantum entanglement. The approach taken by the author is to provide what quantum entanglement is through conceptual examples. There are no wave functions or quantum states discussed in this book. At first, the reader is introduced to two very important concepts in the philosophy of physics; realism and locality. In realism, the assumption is that any physical objects have properties regardless of whether another object with agency (i.e., a person) is observing that object. A typical example of this concept is the following questions:

Does a falling tree in the forest make a sound when no one is listening? 

Realism says yes, it does. In the case of the tree, it has a center of mass that gives the tree some gravitational potential energy that upon falling is converted to kinetic energy and then generates sound waves in the air once it hits to ground. The tree had mass, potential energy, and kinetic energy which according to realism exist objectively. The opposing view is that it was the sound wave came into existence because an agent was listening. This seems absurd and it is in classical physics, but not necessarily in quantum physics.

Locality refers to the fact that observing, measuring, or disturbing objects in a region of a space does not affect other objects at arbitrary distances in that space. Here, I'm using space in an abstract sense not necessarily a Euclidean 3D space. I do note that in the book the discussion of locality is with regard to distances in 3D Euclidean geometry but I think I'm correct in that locality applies to non-Euclidean 3D spaces and this would be the more general statement. Locality is a pretty important concept in physics and is one of the reasons we got the famous EPR paper from Einstein. 

After the book presents these two concepts it gradually moves into the concept of hidden variables, that is properties of objects that can change aspects of the object when observed, but yet the variables themselves are never observed. Hidden variables satisfy realism. Much of the subsequent chapters present examples that lead to the famous Bell inequality which arises due to correlations in probabilities. The bell inequality needs to be satisfied for a theory to contain locality, if it is violated the theory is non-local. As it turns out, at least to our ability to experimental test the theory of quantum mechanics, it is a non-local theory without hidden variables because. All experiments that have been conducted to date violate Bell's inequality and suggest that correlations are instantaneous within the quantum mechanics framework. It should be noted that you could have a quantum hidden variable theory (i.e. Bohemian mechanics) that is non-local which would describe experimental results, but I guess the argument against this is why introduce hidden variable theory if a non-hidden variable theory doesn't provide any additional clarity other than satisfying realism.

It is pretty well documented, or at least we are made to think, that Einstein had serious issues with the non-local (dubbed "spooky action at a distance") behavior of quantum theory as well as the mainstream interpretations not satisfying a realism philosophical perspective. More specifically, the Copenhagen interpretation posits that the wavefunction/quantum state is more of a mathematical tool and is not necessarily a physical object since it only provides a way to extract probabilities of observable properties.

Going back to the book, chapters 3 and 4 provide different and simple experimental setups that look at probabilities and their correlations to arrive at Bell's inequality. The author then reminds the reader that quantum mechanics violates this inequality. Chapters 1-4 are written in a direct and comprehendible manner, but the truth is, I find it actually easier to understand the Bell inequality and violations of it by actually following the simple Linear algebra of quantum theory.  Trying to think through all the words describing the setup and outcomes can become burdensome. Given  the current focus on quantum computing, there are a lot of good books that go through the same results using simple linear algebra. I think it would have been easy to introduce most readers interested in this book to the basics of a qubit, Hilbert space, and corresponding operations, which could help readers understand these concepts more easily.

Chapter 5 goes through the potential inconsistencies of quantum mechanics with special relativity. Personally, I found this chapter was not delivered in the most impactful way, but it addresses the original concerns of physicists.  The final concluding sentence that indicates everything is okay in the end is:

"... the linkage between entangled particles conveys neither mass nor messages"

The author ends the book with a chapter regarding realism and its validity.  I think this is the best section of the book. The author gives their thinking on the topic of local realism by stating:

"The only fact that's (almost) certain is local realism cannot account for measured results"

Thus, local realism is a dead concept in the author's eyes As frustrating as it feels, I would agree with the author. The remainder of the chapter deals with interpretations of quantum theory from philosophical perspectives and you get a nice concrete quadrant table to decide what path to take, namely:

Find falsehoods in assumptions	Abandon locality & realism
Abandon locality & keep realism	Abandon realism & keep locality

I'm not going to go through and explain each of these because I want to leave some excitement, but I think this is the most interesting part of the book. 

I recommend reading this book if you're going to be studying quantum mechanics in any way because it will help with some of the philosophical thinking behind the theory. The reading is extremely accessible to any background and it is very short making for a good weekend read. Here's the book:

MIT Press Store

Reuse and Attribution

Not so hidden: A clearly written popular quantum physics book by a masterful communicator

$^\dagger$My Commentary

First of all, I am a very big fan of Sean Carroll and in general many of the fantastic science and technology communicators in the 21st century. As a society we are in debt to their efforts and time in aiding in understanding the fascinating and intriguing universe we live in. What I enjoy about Sean Carroll's approach to communicating physics concepts and topics is his unmatched clarity in delivery and pace of speech. If you've every listened to his podcast (Mindscape) or others that he has been a guest on, you know what I'm talking about. His most recent popular physics book titled "Something Deeply Hidden: Quantum Worlds and the Emergence of Spacetime" continues his trend of excellence in science communication.

In this book, Sean focuses on the foundations of quantum physics and how we ( the community scientist) have become complaisant with the "shut-up and calculate" mentality when dealing with the quantum realm. The issues stems from the early pioneers of quantum mechanics who could not make sense of the predictions from the mathematics and the experimental observations, namely, that the quantum object/information we call the wavefunction, does not manifest as described by the math when measured in the lab. Lets throw some math into the mix just to make things clear. In nonrelativistic quantum mechanics we have an equation which describes the relationship between the time evolution of the quantum state (i.e. wavefunction) to the evolution of energy content of a system given that quantum state; this is the what we are told the infamous Schrödinger equation:

$$ i\hbar \frac{\partial} {\partial t} | \Psi \rangle= \hat{H} | \Psi \rangle$$

The quantum state function, $\Psi$ is called the wavefunction because in many cases it has a functional form that reassembles wave-like characteristics. This wavefunction differs from our intuitive notion of classical waves in that the amplitude is a complex number.  Sean Carroll's argument is that the wavefunction is all information needed to describe a quantum system. Furthermore, he argues that we take this equation at face value for what it tells us, namely, that given a quantum wavefunction we can describe its evolution deterministically. This is a key statement, because in popular science you may hear that quantum mechanics is a probabilistic theory, that is only true  if we are concerned with knowing additional information about the system using the wavefunction. For example, if we want to know the position, momentum, or energy then we can only speak in terms of probabilistic outcomes of those observable. But the wavefunction always evolves deterministically via the Schrödinger equation. For example if we want to know the probability expectation for observing  or measuring the momentum of a wavefunction describing a particle, we would write something down like:

$$ \langle \Psi | \hat{p} | \Psi \rangle $$

this equation provides us with a mathematical result about the momentum representation of the wavefunction  in a probablistic manner. It is  beyond the scope of my intent for this blog but the reason we can't say that the observed momentum of the wavefunction  $\Psi$ is exact, is related to the fact that the wavefunction is a superposition of equally valid solutions in what is known as Hilbert space. 

Now the main focus of the book is on an alternative understanding for the measurement catastrophy in quantum physics, that is to say, when experiments are performed on quantum systems we don't get the entire probability distribution of the wavefunction for an observable/expectation as an output. What we get is a single data point from that sampling space. If we conduct enough experiments, then of course we recover the distribution. But why don't we get the entire wavefunction probability when we measure? The historical and mainstream thought on this is that something happened by which when an observer (e.g., the human eye or a digital sensor) measured the quantum state/wavefunction so that it "collapased" into a single value. Now you say "What do mean? What forced the function to collapse?", yes this is indeed a strange phenomena. The Schrödinger equation nor any other mathematical interpretation tells us anything about a wavefunction "collapse". For many years I never really thought about this, but more recently it really is bothersome that a quantum state just "collapses" to a single value as if something forced it for which we don't know anything about. Albert Einstein's thought on this was that the forced collapse is due to local hidden variables; things we are unable to identify as being part of the system and are locally causal.

Now Sean Carroll's approach is more epistemic, we know we have this quantum mathematical object called the wavefunction and every thing in the universe can be described by it, so what happens when the quantum system I am describing interacts with an observer who is also treated as being a quantum system. The outcome is that we get  parallel quantum states that are very much deterministic and in existence, but having different probabilistic outcomes (I think this is how I understand it?). In other words, the act of quantum systems interacting produces many outcomes that in a sense occur in parallel worlds, hence the many worlds. To be clear we don't need to think of the same physical space being occupied, but rather that in some abstract representation of many isolated outcomes have occurred with validity. This approach goes by the Everettian or Many Worlds interpretation.

The main tenants of the the many worlds argument are 1.) we should not try and interpret the meaning of the Schrodinger equation but just follow the mathematics as providing what is real and 2.) don't select which systems behave quantum mechanically, assume every physical object in the universe can be described by a quantum state.

Although I very much appreciate Sean's insight and excellent introduction to this foundation quantum physics vantage point, my own human bias doesn't want to agree. Its not that I don't think its a valid understanding of the outcomes of the Schrodinger equation, but more that it leaves me wondering about the other "branches" of the wavefunction. For example, can one roll-back time to traverse a new branch? This should be possible since the time evolution operator in the Schrödinger equation is a unitary operator.

There are two things I should mention, 1.) I'm not a quantum physicist by training so my understanding of the topic could have gone wary, 2.) I haven't finished the book.

$^{\dagger}$ I haven't finished the book yet so this is a partial commentary.

Reuse and Attribution