Terminal Diary Galore

I had this idea the other day about tracking my entire life in the Linux terminal. My thinking is that I have probably typed hundreds, if not millions, of content into the terminal and when I try to recall them I end up doing a Google search¹ to reproduce something I've probably done hundreds to thousands of times. The idea I had of tracking my terminal input/output by recording it, came about when I stumbled accidently upon asciinema and thought "well this might be useful". At first I thought "oh I'll just use this to record workflows that are complicated and I might forget", but then I realized I'll just forget what recording I need to watch.

Then I saw an interesting [post] by Miles Cranmer regarding an approach on how he is using a vector database to make finding and pointing others to github discussions more streamlined. In my head something went off. I was like: "Okay, I think I'm going to record my terminal sessions everytime and feed them into a vector database". Seems straightforward to do so.

Well what use is a vector database really? I mean its just semantic search, and even if I get the retrieved text I'm most likely looking for, will I really remember what I wanted to do with it or what it did. This is where using a LLM would be a tremendous utlity. I basically can have it re-teach me my terminal workflows² and what I did by adding in context, instruction, and descriptions.

So I'm going to try proofing this out. I think the utility of this approach needs to have 2 components:

1. It's easy to start recording and update the vector database.
2. Querying and LLM to find what I'm looking for is straight forward.

For the vector database could use something like Pinecone or Chroma. Then need to regularly update the vector database. Once that is done, I can get the embedding for user input and then query the vector database. This will give me some number of top retrieved results but then the goal should be to use an instructed LLM to always output some useful script or set of steps. Here is an illustration of what I envision:

$ querylog "Show me how to install materials project api/pymatgen, then query the most stable copper oxide structure, and analyze the structure factor (i.e., radial distribution function)"

Which then outputs something like:

To achieve the the desired workflow:


1. In the terminal, create a python venv: python -m venv .venv and activate source .venv/bin/activate
2. In terminal, install materials project api and pymatgen: pip install -U mp-api pymatgen pymatgen-analysis-diffusion
3. To query materials project for Cu-O stable structures in python:


with MPRester(apikey) as mpr:
    # Query the database for all materials containing copper
    copper_oxide = mpr.summary.search(chemsys="Cu-O",
    is_stable=True,
    fields=["structure"])


structure = copper_oxide[0].structure


Get RDF
from pymatgen.analysis.diffusion.aimd.rdf import RadialDistributionFunctionFast
rdf_func = RadialDistributionFunctionFast(structures=structure, 
                                        ngrid=101, 
                                        rmax=10.0, 
                                        sigma=0.2)
r,rdf = rdf_func.get_rdf(["Cu","O"],["Cu","O"])
plt.plot(r,rdf)

The example above is a bit generic and your standard LLM would probably get this correct on its own, but if you had some idiosyncratic way of doing things you would need to prompt the LLM so that it would know those specific details. Here the idea is to just find previous examples and have a generic LLM organize, format, and style in a informative way. Lets see if I can get something working.

Footnotes

---

1. Now its more common for me to use an LLM or something like perplexity. ↩

2. A user on the asciinema site replied to a question post I made, their thinking was to use the .history for capturing all my terminal IO. The issue is that will not capture things that go on in emacs or python interpreter. I really want to capture ever IO that goes on int the terminal. ↩

---

Reuse and Attribution

Acutely Tuned & Karpathy Aware

In the world of AI there is a whole subfield dedicated to making sure AI systems work in the best interest of humanity. From my understanding this is called AI alignment or safety. My guess is that this field researches AI systems by putting them through a serious of tests (in a sandbox) to ensure they behave in the most responsive way. Other than that presumption, I'm not really sure what AI alignment or safet really means.

The reason I bring it up in this post is there was a very interesting [report] posted by Leopold Aschenbrenner¹, a leading AI safety researcher at OpenAI who was fired[1]. In summary the report discusses topics related to energy consumption and security needs for artificial general intelligence (AGI) and artificial superintelligence (ASI) systems, as well as other aspects of national security. My view is that I don't really think AI researchers really know if current neural network architecture will enable AGI or ASI, or what will, but the analysis is quite through and detailed and provides some aspects to think about.

After skimming the report, what stood out to me is the fact that we (the U.S.) would need to improve our energy production and security greatly. I do think Aschenbrenner's analysis brushes aside anticipated improvements in energy efficient hardware for inference. Regardless, there is going to be a huge demand for compute based on AI systems. The other concern is in the global arena, we will be competing with each other and dominence of the world order is on the table. While I'm not so sure the gloom and doom is so pressing, it is something that cannot be ignored.

I suggest reading it even if your not really interested in these types of topics because Leopold, whether you agree with him or not, seems to be very through and brilliant in thought.

Coding GPT-2 from Scratch

I've followed Andrej Karpathy for some time since the release of ChatGPT because he is one of the few people who showed how to build these type of AI architectures from scratch². My plan is to follow Karpathy's 4-hour video and reproduce it using Julia's Flux.jl and Transformers.jl. I'm using Julia so that I avoid copying exactly the same PyTorch code that Karpathy is implementing. I was thinking about using JAX but decided to go with Flux.jl because its more familiar to me at the moment.

Footnotes

---

1. I had never heard of this person but seems very well known in the AI community. Also find him to be very interesting. ↩

2. For whatever reason I am draw to people who build things from first-principles. Yes this is not efficient for useful tools or applications but in my experience it has been really hard to get to a 90% "expertise" level without building the darn thing from scratch. When I was learning Finite-Difference Time-Domain Methods, moelcular dynamics, or density functional theory, I always wrote a basic code from scratch to prove to myself that I knew how the guts worked. ↩

---

References

[1] L. Aschenbrenner, Situational Awareness, (2024). https://situational-awareness.ai/leopold-aschenbrenner (accessed June 10, 2024).

---

Reuse and Attribution

What am I getting wrong here

The implementation of Xie and Grossman's crystal graph convolutional neural network[1] in Julia using Flux.jl and GraphNeuralNetworks.jl [2] is moving along. As I updated in my previous post, I got the data wrangling and processing done correctly. Well, at least I think. It is hard to compare apples-to-apples with the Python implementation and my Julia code for the graph structure. In the original PyTorch version by Xie and Grossman, the graph structure was just conceptual as there were no class objects that specifically represented a graph. Actually, it would be useful if someone familiar with PyTorch Geometric, CGCNN, and Flux.jl/GraphNeuralNetworks.jl [2] could check things out for me. 🙏

But assuming the graph structure of the data is correct, the thing that is even more difficult is the construction of the neural network layers. In the original implementation, the node features and edge features are concatenated into a single feature vector on the nodes, then a convolutional weight matrix (i.e., kernel) is applied to "reshape" the feature vector and update the node features. Let me back up; the concatenated edge features are taken by a summed aggregation over nodes/edges. This means we take the bonds based on some cutoff to neighboring atoms. In the parlance of GNN, I think this is just a message passing scheme. But I'm not entirely sure.

So we have an update that takes the edge features, concatenates them to the node features, then applies a convolutional weight matrix. Then we also include a self-update to the original node feature using a weight matrix. This type of operation updates the node features, but not the edge features. This may make sense because in the original CGCNN implementation, the edge features are represented as radial basis functions. To summarize, this is the equation:

$$ \begin{align} z^{(t)}_{(i,j)_k} &= v_{i}^{(t)} \oplus v_{j}^{(t)} \oplus u_{(i,j)_k} \label{eq:feature} \\ v_i^{(t+1)} &= v_i^{(t)} + \sum_{j,k} \sigma\left(z^{(t)}_{(i,j)_k} \mathbf{W}_f^{(t)} + \mathbf{b}_f^{(t)} \right) \odot g\left(z^{(t)}_{(i,j)_k} \mathbf{W}_s^{(t)} + \mathbf{b}_s^{(t)} \right) \label{eq:update} \end{align} $$

Here we have $t$ being the layer¹ number. The key aspect is that the convolutional weight matrix, $\mathbf{W}^{(t)}_f$, is not globally learned, but rather learned for $i$-th atom environment over $k$ bonds². I'm pretty sure my current implementation does not do this and does what is described in eq. 4 in the original paper:

$$ \begin{align} \mathbf{v}_i\left(t+1\right) = g\Bigg[ &\left(\sum_{i,j} \mathbf{v}_j\left(t\right) \oplus \mathbf{u}_{\left(i,j\right)_k}\right)\mathbf{W}_c\left(t\right) \nonumber \\ &+ \mathbf{v}_i(t)\;\mathbf{W}_s\left(t\right) + \mathbf{b}\left(t\right)\Bigg]\label{eq:Xie_Grossman_eq4} \\ \end{align} $$

The other confusion I have is that the pooling operation appears to be done across each layer update of the node features. This makes sense but I don't think I'm doing it correctly.

Here is my current CGCNN model Flux.jl implementation:

# Define the CGCNN struct
struct CGCNN
    embedding::Dense
    convs::Vector{CGConv}
    conv_to_fc::Dense
    conv_to_fc_softplus::Function
    fc_out::Dense
end 


# Constructor for CGCNN ...


# Pooling function
function pooling(atom_fea)
    return sum(atom_fea, dims=2) ./ size(atom_fea, 2)
end


# Forward pass
function (model::CGCNN)(g::GNNGraph)
    atom_fea = g.ndata[:x]
    edge_fea = g.edata[:e] 


    atom_fea = model.embedding(atom_fea)
    for conv in model.convs
        atom_fea = conv(g, atom_fea, edge_fea)
    end
    crys_fea = pooling(atom_fea)
    crys_fea = model.conv_to_fc_softplus(model.conv_to_fc(crys_fea))


    out = model.fc_out(crys_fea)


    return out
end

The concern is in the pooling operation as you see it's applied only to the final layer update to the node/atom features, which is not what is shown in eq. 2 in Xie and Grossman's paper. The thing though is Zygote.jl which does the automatic differentiation for back-propagation, fails about mutating arrays if I try and store the atom_fea outputs at each Graph CNN layer.

Well, I will keep working through this until I get something that seems to train at the same mean-absolute-error as the original paper. Also, the softmax should be applied to each feature layer, $v_i^{(t)}$ prior to pooling, which I'm not doing (see eq. S1 in [1]).

It would have been nice if a detailed neural network diagram was shown on top of Fig. 1 in Xie and Grossman's paper. I'm probably missing some small detail regarding implementation. The one thing I know is correct is the conv function call because this is just CGConv from GraphNeuralNetworks.jl which is an exact implementation of eq. 5 in [1].

Footnotes

---

1. I think it is easier to think of $t$ as the iteration step because what is happening is the node features are updating at each layer, not the edges though. ↩

2. If you look at Fig. S1 in Xie and Grossman's paper, you see that they actually represent the graph using symmetric bonds and thus two nodes in their representation have multiple edges/bonds. This is why the $k$ index is used. In my implementation, I just build an adjacency matrix and use this to define the graph. ↩

---

References

[1] T. Xie, J.C. Grossman, Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties, Phys. Rev. Lett. 120 (2018) 145301. https://doi.org/10.1103/PhysRevLett.120.145301.

[2] C. Lucibello, other contributors, GraphNeuralNetworks.jl: a geometric deep learning library for the Julia programming language, (2021). https://github.com/CarloLucibello/GraphNeuralNetworks.jl.

---

Reuse and Attribution