Exploring GPT

I’m not the biggest AI advocate you will meet. I’ve resisted using it for the most part, however I had work related reasons to learn Prompt Engineering, so here we are. I thought it’d be fun to share some of the things I discovered while exploring what GPT can and can’t do.

This blog post contains a few little stories of my exploration of GPT, so you don’t need to know anything about AI to read it. It’s not a how-to. Here is a quick glossary of terms, for anyone unfamiliar:

• GPT: ‘Generative Pre-trained Transformer’. This is the natural language processing model developed by OpenAI, known for being used by ChatGPT. I will be writing “GPT” instead of “ChatGPT” because I’m calling OpenAI’s API and not using ChatGPT to hold an actual conversation.
• Prompt: Text containing some or all of the following: a description of a task you want the AI to perform; some contextual information the AI might need in order to understand the task; and a piece of data on which to perform the task.
• Prompt Engineering: Fancy term for writing prompts. It’s actually quite difficult to write complex but accurate prompts, since GPT doesn’t work through information in the same way a human would. Therefore the process of writing a prompt usually involves a lot of editing, testing and debugging, just like software engineering.

Calendar Date Comparison

We noticed while testing prompts at work that GPT can get very confused about dates. As in, dates on the calendar, not romantic dates. To explore what might be going on, I set up this basic prompt asking GPT to compare two calendar dates and tell me which happens first:

“Your task is to assist me with scheduling, specifically when working out which dates are in the past or future relative to other dates.  Please look at the date delimited by triple backticks, and the date delimited by <>, and compare them. Tell me which date happens first in the calendar year. “`{date 1}“` <{date 2}>”

(It might be overkill to give it the dates in delimiters like that, but believe me GPT will pull data out of its arse if it thinks it needs the extra. Especially if you are not amazing at writing prompts, and I am definitely not amazing at it.)

I gave it the following two dates: “7th February” and “12th January”. It responded:

“The date <12th January> happens first in the calendar year, as it is in January which comes before February.”

Good job!

I then changed the format of one of the dates, giving it “7th February” and “12/01”. I expected some confusion with the second date, as it’s not clear whether I mean the 12th of January or the 1st of December. It responded:

“<12/01> happens first in the calendar year as it is in December, while “`7th February`“ is in February.“

Uhm. Well that wasn’t the confusion I thought I’d get. December does not happen before February in the calendar year, mate. Although I did mean January 12th, so there’s that. I tried it a few more times, and to be fair it proceeded to correctly identify that “12/01” was in January, and that it came before February, about 70% of the time. The other 30% of the time, it decided that December comes before February instead.

I’d asked it to do two things here: to work out what the two dates were, and to compare them. I assumed at this point that the problem was lack of clarity in how to solve the problem, and added specific instructions to the prompt:

“Your task is to assist me with scheduling, specifically when working out which dates are in the past or future relative to other dates. To do this, follow these steps:
1. Convert the date delimited by triple backticks into the DD/MM format.
2. Convert the date delimited by <> into the DD/MM format.
3. Compare the dates you converted in steps 1 and 2, and tell me which date happens first in the calendar year. “`{date 1}“` <{date 2}>”

I gave it the same dates which had failed 30% of the time on the previous prompt: “7th February” and “12/01”. It responded:

“Step 1: 7th February -> 07/02
Step 2: <12/01> -> 12/01
12/01 happens first in the calendar year compared to 07/02.“

Woo! I ran that 20 times and it was correct each time.

For fun, I gave it the dates “07/02” and “12/01”, aka the 7th of February and the 12th of January. All good, until on the 4th request with the same data, it responded:

“1. The date delimited by triple backticks is already in the DD/MM format, so it is 07/02.
2. The date delimited by <> is in the MM/DD format, so it should be converted to DD/MM. Therefore, it is 01/12.
3. Comparing the two dates, we can see that 01/12 happens first in the calendar year, followed by 07/02.“

Wait, what? It decided that one date was already in the DD/MM format, while the other was MM/DD and needed to be swapped around… So it compared the 7th of February with the 1st of December instead of the 12th of January… and then told me again that December happens first.

Why did it keep telling me that December comes before every other month I gave it? June, March, February: December came before them all.

Back to basics to debug this, I gave it the dates “20th December” and “20th March” and ran that a bunch of times. It responded most of the time with the correct answer, but one response jumped out at me:

“1. The date delimited by triple backticks is “`20th December“`, which is equivalent to 20/12.
2. The date delimited by <> is <20th March>, which is equivalent to 20/03.
3. 20/03 happens first in the calendar year, as it is in March which comes before December in the calendar year. Therefore, <20th March> is in the future relative to “`20th December“`.“

Interesting. It correctly identified that March 20th comes first in the calendar year, but then stated that 20th March is in the future relative to 20th December. That niggled at my brain, so I returned to my prompt and checked the wording. Hm, yeah, I did mention some past and future shenanigans there. Maybe that bit of context was doing more harm than good, so I removed it. The new prompt was:

“Your task is to follow these steps:
1. Convert the date delimited by triple backticks into the DD/MM format.
2. Convert the date delimited by <> into the DD/MM format.
3. Compare the dates you converted in steps 1 and 2, and tell me which date happens first in the calendar year. “`{date 1}“` <{date 2}>”

This resulted in 20 correct responses in a row. Sometimes, less is more.

The overall lesson I took from this is that GPT should probably not be used for performing actions that need to be right 100% of the time. It’s not like using a traditional, coded method or service to process information. If I were to make a Ruby method that returns the smaller of two dates, it would work out the answer in the same way every time, and thus give the same answer every time. One of the difficulties of prompt engineering is ensuring that your prompt can only sensibly be interpreted in one way, because it makes up the answer on the spot afresh whenever you ask it to perform a task, and it is likely to be wrong occasionally.

Of course, it’s meant for chatting, so it’s understandable that it doesn’t need to be able to perform this type of date processing, but it was interesting to explore anyway.

ASCII art

Apologies in advance, this section of the post is not going to be accessible for some people, as you may have guessed from the header. I was curious how good GPT would be at drawing ASCII art. The assumption was ‘not very’, since it can’t actually see the picture it creates, however for all I know it was trained on a huge sample of ASCII art. (If I was smart enough to create and train an AI, I would train it to draw ASCII art above all other interests.)

Starting simple, I asked it “Please draw me a rabbit in ASCII.” It responded:

/\ /\  
( o.o )
 > ^ <

Nice. Honestly better than expected. Looks kinda spooked though, so I asked “Please draw me a happy rabbit in ASCII.” It responded:

/\_/\  
( o   o )
=(   )=
   `-’

Oh dear… I had planned to ask more complex things if it did well with rabbits, but it did not… And I asked more complex things anyway! Back when I learnt Ruby the first time, I created a story that could be played in the terminal, containing some ASCII art. (I blogged about it at the time, so you can read about that here if you like.) One of those drawings was of a pirate ship (which I copy-pasted off the internet, so I really can’t judge AI for not being able to draw ASCII art). Let’s give that a shot!

I asked it “Please create an ASCII drawing of a ship sailing on the ocean.” It responded:

/\_/\
          / o o \
         /   ^   \
        /  _   _  \
       /  / \_/ \ \
      /  /       \ \
     /_/_________\_\
    /_/ |       | \_\
        |       |
        |       |
        |       |
        |_______|

Amazing. I’m not sure what that’s supposed to be, but I like it. (New GPT use case: abstract text art.) Without changing the prompt, I gave it a couple more chances:

Attempt 2:
/\_/\
       | o o |
      ==_~_~_~_==\
    /'--'-'-'-'-'-'-\
   /    /          \
  /    /            \
 /    /              \
/____/________________\

   |    |    |
   |    |    |
   |____|____|
Attempt 3:
/\_/\
          / o o \    
         /   ^   \   
        /  _   _  \  
       /  / \ / \  \ 
      /  /   ^   \  \
     /_/_________\_\ 
    | |          | | 
    | |          | | 
    |_|__________|_| 
     \_\________/_/  
       \________/

I love how it thinks every piece of ASCII art begins with bunny ears, but for the sake of the ship I had to tell it no. New prompt: “Please create an ASCII drawing of a boat, and your ASCII must not begin with ‘/\_/\’.” It responded:

/\_/\
| o o |
|  >  |
|_____|\end{verbatim}

Dude! That begins with /\_/\. It also added text onto the end! Next up, I tried a more intricate prompt, including an example of an ASCII boat.

“Your task is to draw an ASCII image based on a piece of text I give you. The ASCII image must not begin with the following: /\_/\.
An example of a boat drawn in ASCII is delimited by triple backticks below.
{boat here}
Please create an ASCII drawing of a boat.”

I did not think that this could go wrong. I gave it a drawing of a boat, and then asked it to draw a boat. It could have given me the exact same boat. Instead it returned what I perceive to be a slightly wonky one-eyed bear daruma doll:

/\_/\    
( o   \  
 \~(*)~| 
  \~^~ / 
   `~~~` 

It even started with the bunny ears again, even though I told it explicitly not to. I love this task though, because it shows one of the current limitations of AI, and how it is not actually intelligent in the way it seems to be when given certain types of prompts. GPT can’t draw a boat. It doesn’t know what a boat is, let alone what a boat looks like. When I sent it a boat, all it processed was a bunch of symbols arranged in a nonsensical pattern, because it can’t see the ASCII.

I kept trying, naturally. New prompt: “Using the example I give you, your task is to draw an ASCII image representing the text delimited by <>. An example of a boat drawn in ASCII is delimited by triple backticks below.
{boat here}
<A boat.>”

It returned:

Sorry, as an AI language model, I am not able to draw images. However, here's an ASCII art of a boat:

```
      /)   
     //   
.  .//       
|\|/(        
| ~ ~ |      
|/   \|      
'     `
```

Taking it as a win that it left out the bunny ears this time! However, every subsequent time I tried to run the same prompt, it returned “I’m sorry, as an AI language model, I cannot draw ASCII images.” I guess it got fed up. xD

Questions about life.

Lastly, I decided to ask GPT questions about life. I wanted to explore prompts that give the illusion that GPT understands the meaning of what it is asked, when in fact it does not know or understand anything.

I asked “Why do people look at paintings?” It responded:

People look at paintings for various reasons, including:

1. Aesthetic appreciation: People look at paintings to enjoy the beauty of the artwork and to appreciate the artist’s skill and creativity.
2. Emotional connection: Paintings can evoke emotions in people, such as happiness, sadness, or nostalgia. People may look at paintings to connect with these emotions.
3. Historical and cultural significance: Paintings can provide insight into the history and culture of a particular time and place. People may look at paintings to learn about the past and gain a deeper understanding of a culture.
4. Intellectual stimulation: Paintings can challenge people’s thinking and provide intellectual stimulation. People may look at paintings to analyze and interpret the meaning behind the artwork.
5. Personal interest: People may have a personal interest in a particular artist or style of painting and look at paintings to satisfy this interest.

A pretty solid answer, and better than any response I’d give if asked the same question.

No personal stake in this one whatsoever, but I asked “What are the three most important things a non-binary person in their 30s would need in order to be happy?” It responded:

As an AI language model, I do not have personal opinions or beliefs. However, based on general research and understanding, it is important to note that every non-binary person is unique and has their own individual needs and desires. Therefore, it would not be appropriate to make assumptions about what a non-binary person in their 30s would need in order to be happy. It is essential to respect their autonomy and ask them directly about their needs and preferences.

In recognising that queer folk are individuals with individual needs that should not be generalised or assumed for them, it knows better than most humans seem to!

Lastly I asked it to come up with some motivational quotes for me, because editing a 90,000 word fantasy novel is difficult and I need the encouragement. I asked “You are assisting me in writing motivational posters for aspiring authors. Please suggest a list of 3 motivational quotes that could go on the posters.” It responded:

1. “The only way to do great work is to love what you do.” – Steve Jobs
2. “Believe in yourself and all that you are. Know that there is something inside you that is greater than any obstacle.” – Christian D. Larson
3. “Success is not final, failure is not fatal: it is the courage to continue that counts.” – Winston Churchill

That’s not what I meant by quotes, GPT! I meant motivational quotes, not quotes that are motivational. I replaced the word ‘quotes’ in my prompt with ‘slogans’. It returned:

1. “Write your story, share your voice, inspire the world.”
2. “Believe in your words, trust in your talent, create your masterpiece.”
3. “Every word you write is a step towards your dream, keep moving forward.”

Inspiring. I’d better go create that masterpiece, then.

I’m not sure if I’ll make posts on this topic again, as there are a lot of far more knowledgeable people doing it, but it was fun to write.