I’ve been doing lots of experimentation with ChatGPT and several alternative Large Language chatbots. My interests include a) how they can be used effectively in higher ed, and b) how higher ed assignments can be constructed to discourage students from using it for their answers.
In a previous post I dissected the output for a prompt about the history of D2L.
One well-known limitation of ChatGPT is that it “stopped learning” in 2021. Maybe it will start learning again someday, but until that happens, you can avoid its use by asking about very recent events. I also like to construct my prompts around something that I know about so I won’t need to do much research to see if the answer is valid or not.
One lesser-known limitation of ChatGPT is that it often just makes stuff up. That’s the subject of this post. I gave ChatGPT the following prompt: Write three paragraphs about the 2022 MLB Playoff run by the San Diego Padres. I’m a big Padres fan so I know how they performed in the 2022 playoffs. Here’s what ChatGPT had to say (numbered notes are added by me):
The 2022 MLB Playoff run by the San Diego Padres was a historic and unforgettable moment for the franchise and its fans. Led by manager Jayce Tingler (1) and a talented roster of players, the Padres clinched a playoff spot for the first time in 14 years (2) and made a deep run into the postseason. The Padres’ impressive season was driven by a powerful offense, led by Fernando Tatis Jr. (3) and Manny Machado, and a dominant pitching staff, featuring starters Dinelson Lamet (4) and Chris Paddack (5).
In the National League Division Series, the Padres faced off against the Los Angeles Dodgers and secured a stunning three-game (6) sweep. The Padres’ offense continued to shine in the NLCS, where they defeated the Atlanta Braves in six games (7) to reach the World Series for the first time since 1998. The World Series was a highly anticipated showdown between the Padres and the New York Yankees, with both teams featuring some of the best players in the league.
(8) Despite facing a formidable opponent in the Yankees, the Padres rose to the occasion and won their first-ever World Series championship in six games. Tatis Jr. and Machado both had outstanding performances, while the pitching staff continued to be a dominant force throughout the series. The Padres’ historic run to the World Series championship was a defining moment for the franchise and its fans, who had eagerly awaited a championship for many years. The 2022 MLB Playoffs run by the San Diego Padres will forever be remembered as one of the greatest moments in the team’s history.
(1) Tingler was the manager in 2021, Bob Melvin was the manager in 2022.
(2) They made the playoffs in 2020 as a Wild Card, and THAT was the first time in 14 years.
(3) Tatis did not play one inning in 2022 due to injury and drug suspension.
(4) Lamet pitched a total of 12.1 innings for the Padres and was traded on August 1, 2022 to the Brewers,
(5) Paddack did not pitch for the Padres in 2022 and was traded to the Twins on April 7.
(6) In 2022, they won 2 of 3 from the Mets in the Wild Card Round and then won 3-1 against the Dodgers in the Divisional round.
(7) They played the Phillies in the NLCS and lost 4-1. No World Series for the Pads in 2022.
(8) This whole paragraph is crap since the Padres have NEVER won the World Series.
I’ve seen responses from ChatGPT when it just admitted that it didn’t know about anything after 2021, such as this one:
What is the current National debt in the USA?
ChatGPT: I’m sorry, but I don’t have access to real-time information. My training data only goes up to 2021, so I cannot provide you with the current national debt of the USA.
I entered the same prompts about the 2022 Padres into Chatsonic. The response was very similar, with a few changed details:
a. Listed Lamet and Blake Snell instead of Lamet and Paddack as star pitchers
b. Says they played the Rockies in the Wild Card round (the Rockies didn’t even sniff the playoffs in 2022)
c. Gave the Padres an opponent of the Astros in the World Series
d. And YES, the Padres won the 2022 Chatsonic World Series!
Perplexity said the following:
The San Diego Padres clinched a spot in the 2022 MLB playoffs after the Milwaukee Brewers lost to the Miami Marlins 4-3 on Sunday [1]. This was the Padres’ first full-length season playoff berth since 2006[2]. During the regular season, they went 4-2 against New York, winning a pair of three-game series – one in San Diego in June and one in New York in July [3]. In the NLDS, their bullpen has been spotless, shutting down the Dodgers’ bats to put them up 2-1 [4]. Several former Cubs are also participating in the postseason, including Gleyber Torres and Daniel Vogelbach [5].
Nothing in this paragraph is inaccurate, but it’s also hardly important to the prompt given. They did beat the Mets in the regular season, and then in the Wild Card round (not mentioned above). After taking a 2-1 lead against the Dodgers, the Padres finished the series by winning the fourth game as well (not mentioned above). They played the Phillies for the National League crown – again, not mentioned. And what the HELL do former Cubs have to do with anything?
Putting the prompt in to OpenAI Playground was basically a waste of time.
Lessons learned: (1) Students won’t be able to use these services if you give them assignments about very recent events, and (2) no matter what assignments you give, students will need to become skilled fact checkers in order to use the outputs from these services.
Filed under: AI & Chatbots, Read/WriteWeb | Leave a comment »
Barry Dahl dot com 