Why experimenting with new LLM and generative AI tools is important

Experiment with all kinds of generative AI and LLM tools. You'll find out where the industry is headed and how you can prepare for it.

A few months ago, I wrote about how I experimented with ChatGPT in my writing. At the time, my main conclusion was that large language models (LLM) could not generate text at the level of a professional writer. The tone was off, the writing style was not mine, and the model repeated facts or placed them in the wrong sections. I ended up doing a lot of rewriting. 

However, I also predicted that LLMs would continue to get better and would eventually reach the point where they could become very efficient copilots for different tasks, including writing. Therefore, while I didn’t adopt LLMs as a regular writing tool at the time, I started adjusting my writing process to prepare for what could come in the future. 

In a few months since I wrote, LLMs have continued to progress at an impressive rate. LLM fine-tuning has become a lot more accessible with low-code/no-code tools as well as GPT fine-tuning assistants. 

Million-token context windows are making it much easier to customize LLMs without the need to fine-tune them. New generations of models are becoming increasingly better at following instructions. And I have gathered a rich personal dataset of how I accomplish different tasks, which I can use to enhance LLMs. 

In the past months, I’ve made extensive experiments with different prompting techniques on GPT-4o, Claude 3/3.5, and Gemini 1.5 Pro. And today, LLMs have become an integral part of my writing, helping me considerably with some of my articles.

There are many more complicated ways that I’m using LLMs in writing draft code, debugging software, setting up complex software stacks, and more. But I chose this simple example of writing because it shows how everyone, even those who don’t have a background in computer science or software engineering, can start using LLMs and generative AI.

My main takeaway is this: The LLM space is evolving very fast. Constantly experiment with new tools, models, and techniques. Even if the technology is not in a state to solve your problems, you get an idea of where things are headed and what the technology will manage to accomplish in the future. This will help you start preparing your mindset, your workflow, and your data to take advantage of the current or future generations of models.

That said, here is how I adjusted my writing workflow to take advantage of LLMs. The same process can be applied to any task that has an input and outcome.

An example of using LLMs in writing

I read many papers weekly, and I write several reviews that can have an important impact on enterprise and consumer applications. Previously, my process was to first read the paper and highlight the important parts. I would then review the highlights and write an article, a time-consuming process.

After playing around with LLMs, I started thinking about developing a more systematic workflow that could take advantage of generative AI. I changed my process to copy highlights into a Word document as I read through through the paper. I then sorted my notes into several key sections: challenge, solution, and practical results.

The notes were much easier to navigate as I wrote my article. At the same time, I started developing a dataset of notes and the resulting article that I could later use to customize LLMs.

After a dozen examples, I tried to see if I could get an LLM to write a decent draft review based on my notes. I could then edit the results and add commentary that is not included in the paper or my notes.

I tried several prompt formats with GPT-4, Claude, and Gemini. The format that worked best was to start with a role command:

You are an expert writer of articles about AI research papers. You write in a tone that is easy to read and understand for people who know about tech but are not necessarily AI researchers and ML engineers. I will provide you with my notes from the paper and want you to provide me with a draft. 

Next I included an instruction section that included commands such as the following:

- Use the notes from the paper to write your article

- Make sure the article describes the challenge the paper addresses, the solution it presents, and the results of experiments

- Simplify very technical topics in ways that are understandable for people who have a basic knowledge of AI and machine learning

- Write concise sentences and try to use active voice

- The article should be 800-1200 words

- Never use "utilize.” I hate it. Write “use” instead

And then I started adding in-context learning examples in the following format:

Notes:

[notes from paper]

+++++

Article:

[article text in markdown format]

By adding four examples, I started getting interesting results. Gemini 1.5 Pro in particular proved to be better than the other models in mimicking my writing style. With a 2-million-token context window, Gemini is very good at processing long prompts that include many in-context learning examples. And the Google AI Studio has a very intuitive user interface for creating prompt templates. 

Gemini did a great job of avoiding repeating facts, remaining concise, sticking to the word range, and simplifying some of the technical terms included in the paper notes. The result was not perfect and I still spent a considerable amount of time editing the results. But what I got was a distilled version of my notes plus some added context that was much easier to mold into my writing style.

The main part of my work remains reading the paper and choosing notes. This is especially important because the model can still make mistakes and misinterpret some of the points in the paper. During editing, I revisit my notes regularly to make sure the facts are correct.

Improving LLM applications

I continued to iterate on the prompt. For example, in the initial version of my prompt, I added quotes from the paper and occasional interviews from the authors to the article after editing the draft generated by the model. This required extra editing to make sure the quotes did not repeat facts already mentioned in the article. I started inserting the paper and author quotes to my notes with special prefixes and added instructions to my prompts to use the quotes as is:

- Lines that are marked as "Quote from paper" can be directly quoted and attributed to the writers

- Lines that are marked as "Comment from author" should be attributed to the author interviewed for the article

This further improved the process, reducing the time required to edit the article. As I played around with different AI tools, I found other ways to improve other aspects of my writing.

For example, one of the time-consuming parts of the writing process is getting important quotes from interviews. Now, I have the interview transcribed with OpenAI Whisper, I upload the entire script to Perplexity Pro, and I ask the model to extract quotes for different parts of my article. I then add those quotes to my notes before running my prompt on Gemini.

I will continue to improve the process and the prompt. For example, many papers contain technical terms and concepts that need to be explained to a broader audience. Many of these concepts relate to recent research that is not included in the parametric memory of the LLMs. I am considering creating a prompt pipeline that extracts the concepts that need to be explained from the article. I can then use a retrieval-augmented generation (RAG) system to create draft explainers that can be weaved into the article. 

I will also experiment with fine-tuning LLMs on my dataset. Currently, my input prompt is around 20,000 tokens. I want to see if I can reduce it to just the instructions and the notes for the new article. This can make the process scalable and cost-efficient.

In a previous essay, I discussed how generative AI will affect writing. The essence of the article was that some tasks will be automated, but AI will remain a tool in the hands of creative humans. I still stand by that statement. AI will be a tool in our hands to write better. Sometimes, it will help automate some parts of our writing process. Sometimes—like this article—you’ll want to type in every single word yourself. Humans are the writers. AI is the “writing implement.”

Moving forward

In my conversations with different people, I’m surprised at how few people outside of tech experiment with AI. The main point I’m trying to make in this article is this: Go out there and experiment with AI. Just working with the tools will help you develop a mental framework about where the industry is headed and start thinking about ways to automate everyday tasks in your work. 

Scientists do not agree on how, when, and if AI will make human labor irrelevant. But most people agree that AI tools are becoming increasingly useful and humans who can use AI will surely outperform humans who don’t. 

Please share this article with a friend who thinks AI doesn’t apply to their work.