How to Transition from Traditional Machine Learning to Generative AI — Riding the New AI Wave
Generative AI is undoubtedly emerging as the most exciting field for machine learning (ML) talent. Many engineers who previously worked on traditional ML projects are now considering a move to generative AI.
The market signals are loud and clear: Apple recently announced a strategic partnership with OpenAI to integrate generative AI features into Apple products, just three months after shutting down their entire self-driving car project. This shift highlights the growing potential of generative AI over traditional ML, at least at the business level.
Self-driving cars have long been the prime showcase for AI from a traditional ML perspective, as they use the most advanced computer vision and path planning techniques. While it is disappointing to see Apple giving up on self-driving, it serves as a good reminder for traditional ML engineers to prepare for the new era of generative AI.
What can we learn from the self-driving car revolution?
Prior to working on LLM products at my startup, I dedicated seven years to the self-driving car space. I’m always surprised to see how each tech revolution unexpectedly mirrors its predecessors. The hype around self-driving began around 2014. Back then, most talent in the self-driving car industry had no prior experience in autonomous driving or building cars. Self-driving startups had to draw talent from traditional software or hardware industries, such as mapping, gaming, and mechanical engineering. Despite this, people managed to learn and build self-driving cars together as a team.
Today, we see the hype around generative AI. Similarly, most talent in this space has no prior experience in generative AI or large language models (LLMs). Some may have knowledge of traditional natural language processing (NLP) or ML in general, but LLMs are quite different. Startups are again drawing talent from related industries, such as software development, data science, and infrastructure.
The key here is that generative AI, just like self-driving cars ten years ago, is a highly cross-disciplinary field and attracts talent from many existing industries. It’s important to understand the nuances involved and see how your existing experience can be leveraged in generative AI.
How do traditional ML skills translate Into generative AI?
While generative AI requires new ML skills, the foundational skills in traditional ML roles remain highly valuable. Here’s why:
Math is the king
In the wave of new AI breakthroughs, the importance of fundamental math often gets overlooked. Both traditional ML and generative AI require a strong foundation in linear algebra, calculus, and statistics. These mathematical principles are crucial for understanding complex AI concepts, such as transformers and the attention mechanisms. Without a solid math background, it would be difficult to master any forms of machine learning. The stronger your math skills, the more intuitive you will find working with machine learning models. (Read more here.)
Back to basics with model training and evaluation
The core principles of training models and evaluating model performance are universal across ML domains. Although generative models come with new nuances, the fundamental principles still apply. For example, traditional ML involves metrics such as precision, recall, and F1-score, while generative AI frequently uses BLEU scores for language generation and inception scores for image generation. Regardless of the type of metrics, ML engineers should learn to strike a balance between metrics and model performance.
Data quality is still a prerequisite
High-quality training data is essential, regardless of what types of AI you work on. Traditional ML typically deals with structured data and tasks like feature engineering, normalization, data labeling, and handling missing values. Generative AI, on the other hand, involves handling vast amounts of unstructured text data, with tasks like text cleaning, tokenization, and ensuring data diversity and quality. Despite the differences, both fields require strong data handling skills.
How to transition to Generative AI
Transitioning to generative AI may seem daunting, but I’d recommend starting early because the field of generative AI is evolving faster than before. Here’s how you can start.
Step 1: Start a personal project using LLMs
Rather than taking a full course on Transformers and getting overwhelmed by technical terms, try a quick personal project. Pick something fun and easy to build. Here’s a list of 30 ideas to pick from. The goal here is to experience the end-to-end process by yourself, from model selection to app deployment. Leverage pre-trained models available from OpenAI, Hugging Face, or other platforms. Hands-on projects can provide practical insights and build your confidence.
Step 2: Deepen your understanding
Once you’ve encountered issues or bottlenecks from the experience above, use your insights as the starting point for a deeper study of the model you picked. For example, if your app fails at answering certain types of questions, consider learning how to fine-tune the LLM with a small dataset.
Step 3: Join communities
If you can’t find peers to build LLM apps with you, join online communities or offline meetups. Attend generative AI conferences to network and get inspired. This journey doesn’t need to be alone. Plus, your next job opportunity may arise from these connections. You just never know.
Note for hiring managers
Engineers with existing LLM experience are extremely hard to get. I’d recommend giving a chance to engineers with traditional ML background. Just don’t forget to evaluate the core ML skills mentioned above. Most importantly, evaluate their adaptability and willingness to learn.
Ready to ride the gen AI wave?
Technical trends come and go, but mastering fundamental skills will help you stay relevant and competitive.
If you are in tech but new to LLMs, remember that you don’t need prior LLM experience to jump into the world of generative AI. Focus on the long-lasting skills, and you will thrive in any new era of technology.
