How I Got a Job at DeepMind as a Research Engineer (without a Machine Learning Degree!)

You probably may say (if you don’t know me): “Well, this story is inspiring and all but how did you create your own SE/ML curriculum in the first place?”

Well, before all of this happened, I already had a strong track record of learning on my own. If it still feels like magic I strongly recommend you go ahead and read this blog:

I’ll leave a TL;DR here: through my workout routines, learning human languages throughout my life, and maths (all of that on my own) I got skillful at learning. Coursera’s “Learning how to learn course” helped as well.

Now that you have all of the necessary context let me tell you about the ML curriculum I followed to land a job at DeepMind!

Part 2: ML curriculum (January 2020 — June 2021)

New Year 2020 came along. I just finished reading “Crushing it” from garyvee and decided I should start my own YouTube channel sharing my ML learning journey along the way.

The pandemic struck and I was in warp speed again. The stars aligned. ⭐

I decided that over the next year I want to cover various subfields of AI, and I roughly decided that I want to dedicate 3 months to each one of these.

But things weren’t that smooth in the beginning, I was still figuring stuff out. My first subfield, neural style transfer (NST), took longer than 3 months because I found it enjoyable and I thought why rush it?

Along the way, I found a perfect strategy for myself.

I structured my learning the following way. I had “macro” cycles (3-month periods where I tackle a new subfield) and I had multiple micro-cycles interspersed over macro.

The micro-cycles are of 2 types:

• Input mode: I’m ingesting information. The goal is to either get a high-level understanding of the structure of the subfield (blogs, videos) or an in-depth understanding of the topic at hand (research papers, books, etc.)
• Output mode: I’m sharing information I accumulated during my input mode. Teach others! Create public artifacts like YouTube videos, GitHub projects (1 project in the middle of macro), and blogs (1 blog at the end of a macro). As well as higher-frequency updates on LinkedIn. Much later I started sharing on Twitter and Discord as well.

Now the tricky part was combining all of this with my full-time job at Microsoft! It took a significant amount of willpower. I’d hit the program as soon as I wake up for 2h, I’d go for a quick stroll, I’d do my job at Microsoft and a 30-minute power nap after I wrap it up, and finally I’d be working for additional 2–3 hours before going to sleep. The pandemic helped me maintain this crazy rhythm.

The time management and stress handling probably deserve a separate blog post, but it mostly boils down to willpower, correct mindset, as well as taking smart breaks (power naps are golden — let me know in the comments if you’d like a separate blog on the topic of time management).

Now let me go into the specifics of the macros I did on my way to DeepMind! Luckily I started writing blogs sometime around my transformers macro so if you want to know all the nitty-gritty details and the strategy I took, I’ll link the blogs in every single macro section.

For the first 3 macros (NST, DeepDream, and GANs) I, unfortunately, didn’t write a dedicated blog, but the intro section of the transformers blog below as well as the YouTube videos and projects I created can fill in the gaps:

Having said that, here are the TL; DRs of every macro I did:

[1] NST (Neural Style Transfer)

During this period I was learning about neural style transfer. I ended up reading a pile of research papers (though a lot less than later when I became proficient), implementing 3 different projects, which I then open-sourced on my GitHub, and I made an NST playlist on YouTube.

I think that this is a great topic to start with as it’s highly visual and visceral. I recently wrote some of my thoughts on a similar topic in this LinkedIn post. Also, digital image processing and computer vision were the fields I felt the most comfortable with and so that’s why this was a great starting point for me.

I perfected my PyTorch knowledge, learned a lot about CNNs and optimization methods, improved my presentation skills (just check out those README files 😂), became more proficient at reading research papers, and improved my SE skills in general — to name a few.

Here is an NST image synthesized using the code I wrote:

[2] DeepDream

Ever since I first saw the pictures created with DeepDream I was fascinated. I knew I had to learn it. Every single detail. I couldn’t just use online generators and not understand what was going on.

I was reading blogs, analyzing DeepDream subreddits, and exploring various codebases. Most of the original code was written in Torch & Lua combo. I remember losing a lot of time trying to set it up on Windows. Then I switched to Linux, got it to work, only to realize that I won’t be using that repo either way. 😂

My output was a single video on the theory behind DeepDream and an open-source project. I learned a lot by doing this and I enjoyed the process. I used my code to generate these beautiful images, among them the current visual identity of The AI Epiphany. Here is an example:

[3] GANs (Generative Adversarial Networks)

GANs were still very popular in early 2020. I felt like I’m missing the necessary background and so I decided to tackle them next.

I’ve read all of the seminal papers (as well as other less prominent papers) and I decided to implement vanilla GAN, cGAN (conditional GAN), and DCGAN models. Here is an output from the DCGAN model I trained:

At this point in time, I refined my strategy. I realized that:

• I need to write a blog at the end of each macro, summarizing what I’ve learned. That will help future Aleksas (and Aleksa from the future) tackle topic X.
• I need to cover papers on my YouTube channel as I’m reading them. A similar thought process — by doing this I’ll learn better and I’ll help others.
• I need to open-source a project in the middle of the macro. After you implement something you understand it way better — that was/is my thinking. After that, all the papers I read made more sense.

Unfortunately, I don’t have GAN paper overviews from this period. Back then I was experimenting with videos such as “PyTorch or Tensorflow” which turned out to be the most popular ones (proving a hypothesis I had in my mind).

But I didn’t seek popularity (not that I would mind in general), I was seeking a highly relevant audience. I’d much rather have 1 guy/gal from “DeepMind” following my work than 100 beginners — because I knew what my goals are.

[4] NLP & transformers

I knew I wanted to learn more about BERT and the GPT family of models. Transformers were everywhere, and I didn’t quite understand them.

This time I did everything correctly and executed the plan I sketched above. I ended up implementing the original transformer paper from scratch and learned a ton! I decided to create an English-German machine translation system. Since I speak those languages, I thought that’ll make debugging easier, and it did.

I’m so grateful I did it back then — every now and then you need to understand the QKV attention, and it’s second nature to me now. You can find out much more about this journey here:

[5] Graph/Geometric ML

Even before I started doing transformers I knew I wanted to dig deeper into Graph ML (before it became as popular as it is nowadays).

And, in retrospect, I’m very glad I covered transformers/NLP first since the field of NLP had a huge influence on this field. GAT was directly inspired by the original transformer paper, DeepWalk by Word2Vec, and so on.

At this point in time, I was already chatting with Petar Veličković on a fairly regular basis (I’ll tell you a bit later more about how this came to be) and he knew that I’ll be covering GNNs in the next period. He was very receptive and he told me that I can ping him whenever I felt stuck!

I realized that this is a great opportunity! I’ll be learning Graph ML, a topic that fascinated me and I’ll be communicating with Petar, a researcher at DeepMind, who was among the best researchers in this particular field.

Needless to say, I learned a lot! I made a “popular” YouTube GNN series that was shared by influential Graph ML researchers like Michael Bronstein, Petar, and others.

I also made a popular PyTorch implementation of GAT (Petar is the first author of GAT). The reason it became so popular is that it was, according to others, the most beginner-friendly resource out there. With 2 Jupyter notebooks, PPI and Cora datasets support, a nice README, and accompanying videos it filled the gap and made it easy for beginners to enter the field.

(It became a recommended repo for a Cambridge lecture on GNNs and I even got citations. 😅)

I again massively used the fact that I could ping Petar if I had any problems and that this could be a great collaboration between the 2 of us!

For the whole story check out this blog:

[6] RL (Reinforcement Learning)

Drum roll. Surprise, surprise! I really wanted to learn RL either way but it so happens that DeepMind is famous for its RL breakthroughs. 😅

There were so many papers and projects I wanted to understand! Like AlphaGo, DQN, OpenAI’s robotic hand that could solve Rubik’s cube, etc.

This field proved to be very different compared to other ML subfields. The data point independence assumption doesn’t hold anymore, a random seed can make your agent fail to converge, etc.

I’m very glad I already had a decent amount of experience before tackling RL, otherwise, I might have gotten demotivated. I started with stuff that’s very close to my heart — computer vision — and I used it to ramp up my ML knowledge. Then, I slowly progressed toward topics I knew little about.

I again had some help here! Petar connected me with Charles Blundell (who’s also a DeepMinder) and so from time to time, I used that opportunity, although a lot more frugally.

For the whole story check out this blog:

‏Source gordicaleksa.medium.com