Before I start, some of you may have come to this platform for the first time as my link brought you here and may be wondering why have I chosen to publish this article here. For a start [it is an easy platform to use ](https://www.fxhash.xyz/article/a-review-of-fx\(text\))once you familiarise yourself with basic use of blockchains. Second, I wanted to timestamp this article as I believe it could be interesting for me to record the progress of these LLMs for myself. To me, an obvious use case for a blockchain is when one wants to publicly record something with a timestamp in an immutable way.  Third, FxHash smart contract and IPFS notwithstanding, I own my own material here. Fourth, I just wanted to try it out, ok?

In the Google SRE book (are you jealous of my rock 'n roll life?) there is reference to a fictional service: Shakespeare.google.com - a service that is supposed to count the number of instances of a word in the works of Shakespeare.

I decided to try a small experiment to check how a couple of different LLMs fared in performing the task of that fictional service. I wanted a word which I know occurs but of which there might be identifiably few instances so I settled on "tennis", from one of my favourite monologues in one of my favourite Shakespeare plays, Henry V.

What follows is my conversation from ChatGPT (3.5) and Gemini as well as a straight Google search:

**ChatGPT**

User

*how many times does the word "tennis" appear in Shakespeare's plays?*

ChatGPT

*The word "tennis" doesn't appear in any of Shakespeare's plays. Tennis wasn't as popular or widespread during Shakespeare's time, so it's not surprising that it doesn't show up in his works.*

User

*Please check the play Henry V*

ChatGPT

*I apologize for the confusion. Let me double-check the text of "Henry V" for occurrences of the word "tennis."*

User

*are you still checking?*

ChatGPT

*I apologize for the delay. Upon checking, it appears that the word "tennis" does not appear in William Shakespeare's play "Henry V." My apologies for any inconvenience. If there's anything else you'd like to know or if you have any other questions, feel free to ask!*

User

*That is wrong - please check the full text of the play Henry V by Shakespeare to find me an instance of the word "tennis"*

ChatGPT

*I'm sorry for the confusion earlier. Let me check the full text of Shakespeare's "Henry V" for any instances of the word "tennis." It may take a moment.*

User

*how many times does the word "tennis" appear in Shakespeare's plays?*

ChatGPT

*The word "tennis" appears only once in Shakespeare's plays, specifically in "Henry V," Act 1, Scene 2, where it's mentioned by the Dauphin of France in a mocking tone towards King Henry V of England. The line is:*

*"Will your answer serve fit to all questions?"*

*"You can ask any quite-gently, and answer them fair, 'how the tennis-court may be?"*

User

*what about the line "tennis balls my liege"?*

ChatGPT

*Apologies for the oversight. You're correct. The line "tennis balls, my liege" is indeed from Shakespeare's "Henry V," Act 1, Scene 2. It's part of the Dauphin's mocking gift to King Henry V, referring to the tennis balls as an insult. Thank you for pointing out the correct line!*

\*\*

Now **Gemini**

User

*how many times does the word "tennis" appear in Shakespeare's plays*

Gemini

*The word "tennis" does not appear in any of William Shakespeare's plays. Tennis, while a popular sport in England during Shakespeare's lifetime,  predates the invention of the modern tennis ball. The sport referred to as tennis back then was more akin to handball. The term "tennis" entered the literary world much later.*

User

*Please check the play Henry V*

Gemini

*You're right, I apologize for the mistake. While "tennis" wasn't commonly used in Shakespeare's time for the sport we know today, there seems to be some confusion about its usage in Henry V.*

*According to my current knowledge, the word "tennis" does not appear in Henry V.  However, there's some scholarly debate about the word "balls" appearing in the play multiple times, which might be a reference to an older form of tennis.*

User

*what about the line "tennis balls my liege"?*

Gemini

*I see! You're absolutely right.  There is a famous line in Henry V, Act I, Scene 2, where the messenger Exeter presents a box to King Henry and announces "Tennis balls, my liege."*

*This line is indeed a reference to the sport,  but it's used in a rather insulting way by the French Dauphin.  He's sending tennis balls as a mocking gift, implying that the king of England is immature and unfit for war. King Henry delivers a powerful speech in response,  turning the insult on its head.*

*My previous response contained outdated information, and I apologize for the confusion.*

And finally, a **Google search** at least recognises and returns the excerpt in a scholarly article:

![](ipfs://QmRQpdryWZvS22QHsmsT2m1dTHuRwYTY2TsrCxYgt7Sg1m)

I still haven't got an answer though. I believe it must be between 1 and 2, depending on whether you count "tennis-court" as an instance or not - a reference I had not remembered.

It is very interesting how definitively both LLMs respond initially. While there may have been skepticism to begin with we have already come to a point where people are beginning to use the LLMs instead of search - and yet in this instance the Search was "less wrong". At least I knew it had not answered the question. Bear in mind we are not talking about a nuanced opinion here - this is \*simply\* a count of a specific word in a well known text. I hammered the LLM to eventually get a fact that I already knew.

Is this to do with context window sizes? Is it because not all works have been indexed yet? Is it to do with the cost of tokens cs the cost of running a search? When the next person asks this question will the answer be correct? I don't know yet. The number of words in Shakespeare's works are [around 885K words](https://www.folger.edu/explore/shakespeares-works/frequently-asked-questions/#:~\:text=How%20many%20words%20did%20Shakespeare,884%2C647%20words%20and%20118%2C406%20lines.) apparently. This a relevant stat for me because it may have implications for companies wanting to index their entire knowledge.  Putting an LLM on top of previously siloed documents, reports, customer conversations  along with externally  data to quickly access this information in a conversation-style format is an obvious use case. Implementing it effectively may not be trivial however.

Next steps:

1\) Ask someone else to run the same search and record here.

2\) Download the full works and try out different embedding models, chunking ranges, DBs and anything else I learn about on this journey.

I will add to this article if and when I have more useful information.


A recording of responses from a couple of LLMs and google search

State of GenAI

Written by

Editions

Share

Details

stay ahead with our newsletter