What is the ChatGPT plugin? The next phase of conversational AI has arrived (2023)

Chat-GPTOne offastest growing appThis is causing many tech companies to rethink their business models. So far, however, OpenAI's chatbots have generally pulled knowledge from their own internal servers.

meetPlug-in(opens in new window)third parties such as Instacart or Kayak may connect to ChatGPT using add-ons that display information from these companies. It's like the app store version of ChatGPT, an Amazon Alexa skill, or a Chrome browser extension.

the firstannounced in MarchStarting May 19th, anyone with a ChatGPT Plus account ($20 per month) can use the ChatGPT plugin.

OpenAI announced that ChatGPT Plus users can now enable the plugin in settings. (Source: OpenAI Discord)

"The plugin allows companies to easily integrate company-specific context into the range of possible actions ChatGPT can take," said software developer and owner Daniel Galati.Become a better developer(opens in new window)"The types of use cases that this feature enables are virtually endless."

Access to plugins was previously restricted towaiting list(opens in new window)Making it accessible to the (paying) masses has the potential to shape the future of chatbots and web search, although it may be OpenAI's first big flop. (How often do you use your Amazon Alexa skills?)

What are ChatGPT plugins and how do they work?

Any company, individual (with technical skills) or OpenAI itself can develop plugins and make them available to ChatGPT users.

OpenAI currently provides two plugins. FirstProvide ChatGPT internet access, which is required for conversations about current events. The core AI training data is only available until September 2021. Anyone using the free public version of ChatGPT will encounter this limitation, as was the case with PCMag back in the dayComparison of ChatGPT, Google Bard and Microsoft Bing AI.

Matthias Keller, Kayak's chief scientist and senior vice president of technology, said the web browser plug-in allows users to ask timely questions, such as "How did the Celtics play last night?" website, scrapes the data and extracts the results.”

OpenAI's second plugin, Code Interpreter, helps users perform complex calculations in the Python programming language. "Having access to a very enthusiastic junior programmer working at fingertip speed can make a whole new workflow easy and efficient, and demonstrate the benefits of programming to a new audience," says OpenAIexplain(opens in new window).

Code interpreter with text overview (back) and popup box with calculations. (Source: OpenAI)

Third-party plugins, such as those developed by the Keller team, work differently. They connect ChatGPT to external companies like Kayak or Expedia to help users plan trips, or use Instacart to shop for groceries.

In the viral tweet below, a ChatGPT user activated the Instacart plugin and asked to create a meal plan. He provides a budget, a quick breakfast, and a favorite dish for dinner (Pakistani, Chinese, or Italian). ChatGPT then created a list of recipes and provided a link to add all the groceries to the Instacart cart. For existing Instacart users, the plugin will automatically select the most recently used grocery store (or the most popular grocery store for new users).

tweet(opens in new window)

Kayak or Expedia plugins are best for generating ideas: “Suppose you want to travel to Europe this summer but aren’t sure when. Or maybe you want to find the cheapest place between Madrid or Barcelona. Kayak has all that information, but it brings it to the surface," Keller said.

Too much power? plugin problem

Like anything related to ChatGPT, plugins are controversial. First, they are concentrating more power, money, and influence in the hands of OpenAI by directing people to interact with ChatGPT instead of independent sites and companies. that's a problemPCMag-HighlightsIn the creative industries – where chatbots take over web traffic and thus generate revenue from the source – but for all types of companies and organizations.

In the kayak example, there isAndThere are still people who want to use the main website instead of chatting with ChatGPT. "If you know you're going to New York City for business next week, you might just want to go to the Kayak website and book your flight and hotel," Keller said. He pointed out that ChatGPT can only extract a fraction of the results people see on Kayak.

tweet(opens in new window)

In the long term, companies can do without the ChatGPT plugin when developing similar services. "I think there are some major flaws in the plug-in approach," says Chris Watts, Founder and CEO of CoFounder Inc. "Companies don't want to be fully involved because it means a completely different user experience that takes users out of the context of their own application.”

He's right. Instacart is developing its own AI assistant - "Ask Instacart" - which will appear in the app and launch later this year. Expedia, which offers the ChatGPT plugin, does the same. Both use the underlying technology of ChatGPT,GPT-4to perform backend tasks while the user stays on the company website.

"The first is an Expedia plugin available on the ChatGPT website that allows users to start a conversation within ChatGPT to plan a trip," Expedia told PCMag. "The second exciting product changes that by integrating ChatGPT into our Expedia iOS app."

But for small businesses, the plugin can bring AI to companies and services that don't have the resources to build their own tools. "External companies can leverage this technology to connect their businesses and users with minimal effort," said Galati of BeABetterDev. "It's a paradigm shift like no other."

Limitations of the ChatGPT plugin

ThereAndThere are some things that plugins cannot do, such as processing payment information or fulfilling orders. ChatGPT may offer the best shopping list or itinerary in Barcelona depending on your budget, but people still have to complete the process on these companies' respective websites. Speaking of the website, ChatGPT does not have a mobile app, so this experience is limited to desktop.

"Currently, ChatGPT is still linked to Kayak," Keller said. "Currently there are no transactions within ChatGPT, no payment notifications, addresses or similar details."

This is an advantage for Instacart and Expedia, which have developed chatbots on their websites, even if those chatbots are based on OpenAI technology. It simplifies the experience for users who have the same on-site experiences as ChatGPT, with the added benefit of being able to complete transactions like booking travel or ordering groceries.

Another limitation of ChatGPT: users can install as many plugins as they like, but only use three at a time. And you can only activate one plugin in a category, such as KayakvonExpedia for travel planning, says Keller.

What ChatGPT plugins are available today?

tweet(opens in new window)

In addition to OpenAI's two plugins (web browsing, code interpreter), the company says70+ 3rd party plugins(opens in new window)Already done, the first ones are listed below. They cover a range of industries and businesses including shopping, real estate, stocks, travel and groceries.

• www.expedia.nl-Website

• tax notice

• Instant shopping cart

• Kayak

• Klarna

• Miroku

• open table

  (Video) How to Access and Use NEW ChatGPT Plugin: Huge AI UPDATE! (Code Interpreter)

• Shopping

• hit

• say

• Wolfram

• Zapier

Others are more concerned with the productivity of labor use. Two - AskYourPDF and ChatWithPDF - ask the user for a PDF document URL, which ChatGPT searches for information, Mashablethis report(opens in new window). This is useful for particularly long documents such as contracts (but be sure to double-check the output). GitHub also maintains aThe list(opens in new window)The most useful plugin for encryption.

How do I create a ChatGPT plugin?

OpenAI designed its infrastructure so that anyone with the right skills could create a pluginFollow these instructions(opens in new window). People are definitely trying. The OpenAI Discord channel has an entire section called "Plugin Showcase" where people can showcase their new creations. One called TextbookGPT introduces academic charts to visualize student questions.

Example of a TextbookGPT plugin. (Source: OpenAI Discord/user@lessand.ro)

Another app called Shifu connects ChatGPT to IoT devices. "With this plugin, ChatGPT can not only extract data from devices, but also control them," said its inventor @Yongli. "By connecting a webcam via a plug-in, for example, users can view live images and control the camera in a chat conversation, opening up new possibilities for AI-human interaction."

Shifu plugin demo on OpenAI Discord. (Source: OpenAI Discord/Shifu Creator-Gebruiker @Yongli)

To create Kayak's plugin, the Keller OpenAI team provided two key pieces of information: how to access Kayak's existing API and documentation explaining the information in the API.

"We used an industry standard document called OpenAPI, which is often very technical," he said. "What we're doing for OpenAI is enriching it in human-like ways,natural languageNote. [For example] if we had a field name like Airport of origin, we would explain in a sentence or two that this is where a person wants to start the flight. "

These details make the data in Kayak's API more chatty and help ChatGPT better recognize when you're chatting with other people.

The process is relatively straightforward, Keller said, and sets the standard for how developers expect plugins to work on top of other AI tools like Google Bard.

"What's special about it is how easy it is for developers like us," he said. “Let's say you want to develop [skills] for Alexa. You need to do a full integration where you download Amazon's SDK, block your data in the right places and it starts flowing. [But the way OpenAI integrates] you basically show. That's it for your documents.

Is a ChatGPT plugin available in the free version?

Currently only paying users can explore the plugin. However, this may change with OpenAIApproach to introducing three-phase functions(opens in new window).Plugins was previously in an alpha stage "to give a small group of users ... access to new features for testing and feedback."

OpenAI said the feature will be released in beta to all ChatGPT Plus users on May 19 to provide a larger testing and feedback group. After OpenAI assesses "whether the feature is ready for a wider audience," they can make it available to all ChatGPT users.

Is there a plugin for Google Bard?

The upcoming Google Bard extension (Image credit: Michael Kan)

Google Bard doesn't offer a plugin yet, but it does at last week's I/O developer conferenceGoogle says it plans to add themin the near future. Like OpenAI, Google will develop its own plugins and offer others from third-party providers such as Walmart, Spotify, and Uber Eats.

"In the future, we'll be bringing features from Google apps and services that you may already be using — Documents, Drive, Gmail, Maps, and more — into the Bard experience," wrote Sissie Hsiao, vice president and general manager of Google Assistant and Bard style in oneblog entry(opens in new window)"With third-party extensions, Bard will also be able to leverage the services of the Internet, allowing you to do things that weren't possible before."

Google lists similar partners for its plugin as ChatGPT: Kayak, OpenTable, ZipRecruiter, Instacart, Wolfram andKhan AcademyKeller said Kayak hasn't received any information from Google about when it can start building plugins because the product is still in development.

From what Google says, it's starting to use a more intuitive, image-based plugin. forthcomingIntegrate with Adobe's AI productsGenerate images in Bard that resemble those ofgive herexisthalfway through the journey.

"Let's say I'm throwing a birthday party for my 7-year-old who loves unicorns and I want to send a cute picture with the invitation," Google said. “All I had to do was ask Bard, 'Take a picture of a unicorn and a cake at a children's party' - and the picture was created in seconds while adhering to Adobe's high quality standards and ethical responsibility. "