Microsoft outlines framework for building AI apps and copilots; expands AI plugin ecosystem - Source (2023)

Remember when software wasn’t connected to the internet? Didn’t think so. In a handful of years, this will also be true for software that doesn’t come with an intelligent copilot assistant, according to Kevin Scott, Microsoft’s chief technology officer.

A copilot is an application that uses modern AI and large language models to assist you with a complex cognitive task – from writing a sales pitch or catching up on a missed meeting to generating images for a presentation or planning a themed dinner party.

Microsoft introduced the concept of a copilot nearly two years ago with GitHub Copilot, an AI pair programmer that assists developers with writing code. This year, Microsoft rolled out copilot experiences across its core products and services, from the AI-powered chat in Bing that’s changing how people search the internet to Microsoft 365 Copilot, GitHub Copilot X, Dynamics 365 Copilot, Copilot in Microsoft Viva and Microsoft Security Copilot.

Today at the annual Microsoft Build developers conference, Microsoft announced that it has expanded this ecosystem of Microsoft Copilots to include Copilot in Power BI and Copilot in Power Pages in preview, Copilot in Microsoft Fabric, available in preview soon, and Windows Copilot, which will start to become available for preview in June. The company also introduced new features that will help developers build their own copilots and next-generation AI applications. This includes new tools called plugins that make copilots more useful by allowing them to interact with other software and services.

“You may look at Bing Chat and think this is some super magical complicated thing, but Microsoft is giving developers everything they need to get started to go build a copilot of their own,” Scott said. “I think over the coming years, this will become an expectation for how all software works.”

Expanding the plugin ecosystem

OpenAI introduced ChatGPT plugins in March. Microsoft announced plugins for Bing earlier this month. That technology allows ChatGPT and Bing Chat to help you find and book a restaurant reservation using an OpenTable plugin, for example.

Today, Microsoft is adopting the same open plugin standard that OpenAI introduced for ChatGPT, enabling interoperability across ChatGPT and the breadth of Microsoft’s copilot offerings. That means developers can now use one platform to build plugins that work across both business and consumer surfaces, including ChatGPT, Bing, Dynamics 365 Copilot, Microsoft 365 Copilot and Windows Copilot. Microsoft also announced it is bringing Bing to ChatGPT as the default search experience.

As part of this shared plugin platform, Bing is adding to its support for plugins. In addition to the ones previously announced for OpenTable and Wolfram Alpha, it will also have Expedia, Instacart, Kayak, Klarna, Redfin, TripAdvisor and Zillow among many others in the Bing ecosystem.

In addition, developers will now be able to extend Microsoft 365 Copilot with plugins. Plugins for Microsoft 365 include ChatGPT and Bing plugins, as well as Teams message extensions and Power Platform connectors – enabling developers to leverage their existing investments. And developers will be able to easily build new plugins with the Microsoft Teams Toolkit for Visual Studio Code and Visual Studio.

How plugins work

At Microsoft Build, the company showcased an AI development framework that helps developers build their own copilot. This stack includes support for plugins that augment the capabilities of AI systems by allowing them to retrieve real-time information, incorporate company or other business data, perform new types of computations and safely take action on the user’s behalf.

At root, think of plugins as a bridge. This could be a bridge between a large language model that was trained on public data from the internet and all the data that a company may keep privately about its benefits. The plugin is the bridge that gives the copilot access to those files when it answers a question from an employee at the company.

Similarly, a plugin could serve as a bridge between a large language model and a website or back-end system that a company uses to book business travel, enabling the copilot to make arrangements for a salesperson’s trip to San Francisco that are in line with the company’s travel policy.

“A plugin is about how you, the copilot developer, give your copilot or an AI system the ability to have capabilities that it’s not manifesting right now and to connect it to data and connect it to systems that you’re building,” Scott said. “I think there’s going to eventually be an incredibly rich ecosystem of plugins.”

Expanding the plugin ecosystem

Developers will also be able to create, test and deploy their own plugins in a number of ways – to eventually deploy for use with Microsoft Copilots and to augment the capabilities of their own applications built with generative AI technology.

For example, a developer at a large corporation may want the Microsoft 365 Copilot to be able to access the company’s contracts with customers and vendors in order to ask questions about how certain legal issues were handled in the past. To do that, the developer needs to connect the Microsoft 365 Copilot with the private database of legal files, which requires the documents to be encoded, indexed and stored in such a way that they can be searched.

“That’s a pattern of how you would load information into the model,” said John Montgomery, Microsoft corporate vice president of program management for the AI platform. “And that’s where plugins come in. What a plugin does is it says ‘Hey, we want to make that pattern reusable and set some boundaries about how it gets used.’”

Microsoft is releasing a set of capabilities to facilitate the creation of plugins that work across its copilot surfaces. Visual Studio Code, GitHub Copilot and GitHub Codespaces will make it simple for developers to create, debug and deploy new plugins, for example, and Azure AI will add capabilities to run and test plugins on private enterprise data. Once created, these plugins will work across Microsoft’s Copilot experiences.

Montgomery noted that the addition of plugins will accelerate the pace at which Microsoft customers can develop innovative applications with Azure OpenAI Service, which provides access to OpenAI’s generative AI models including GPT-4 and is currently serving more than 4,500 companies.

“What you’re going to see at Build is our current best thinking about the architectural practices, the tooling you need, and the app patterns and user experience designs you want to be putting around these new copilot applications,” he said in an interview prior to the conference.

New tools for building your own copilot

While Microsoft has deployed generative AI technology into the company’s products and services, developers will eventually build most of the copilots in the world, according to Scott.

“They will understand a particular thing that they or their users are trying to accomplish, and they will use this AI software development pattern to go build those things for those users,” he said.

Copilot development begins like any software project: as an idea. What’s different is the realm of possibilities that this AI development platform enables, Scott said.

He likens this to the new experiences and services that appeared with the arrival of smartphones. Many things that were impossible to imagine became possible. Some were straightforward to develop, such as apps for familiar, casual games. Others were hard and harnessed the full capabilities of the smartphone platform to create new types of experiences, including services that disrupted the taxi and vacation rental industries.

“They went from impossible to hard and because they were hard and really used the capabilities of the new platform, they found enduring success,” said Scott, alluding to his expectation that similarly disruptive applications will be built with the copilot stack.

With a solid idea for an application in hand, developers can go about turning it into reality, he said. The process starts with selecting a generative AI model such as GPT-4, which is available through Azure OpenAI Service. The next step is to give the copilot a meta-prompt, which is a base description of the copilot’s role and how it should operate. In addition, the copilot needs memory to keep track of a conversation with a user and respond with appropriate context and awareness. Finally, the copilot needs the ability to access data and other services, which plugins can provide.

Copilots deployed to support complex tasks, such as writing a whole book, will also require orchestration – the ability to break down tasks into multistep components and execute the tasks according to a generated plan.

Developers can use tools such as the Semantic Kernel software development kit that Microsoft released to open source in March to integrate large language models with conventional programming languages. The kit includes tools for memory and orchestration and support for plugins, which accelerates the development of copilots, according to Montgomery.

At Microsoft Build, the company also introduced new tools to simplify the development of copilots. For example, with the new Azure AI Studio, developers can now more easily ground conversational AI models on their private data. With a new capability called Azure OpenAI Service on your data, developers can uncover organization-specific insights from data, text and images using natural language-based application interfaces. To further extend the capabilities of large language models, Microsoft also announced support for plugins with Azure OpenAI Service. Azure AI also now supports Azure Machine Learning prompt flow, a sophisticated prompt engineering tool, noted Montgomery.

“We’ve built a bunch of these copilots. We’ve looked at a bunch of the different architectures out there for how you load other information in, how you create reproducible content, how you feed the model the right stuff so that it gives the right answers,” he said. “Build is going to be a celebration of all the things we’ve learned, the tooling we’ve created and bring that out to the world.”

Responsible AI updates

Importantly, developers also need to ensure the copilot returns the intended results and avoids outputs that are biased, sexist, racist, hateful, violent or prompt self-harm, noted Sarah Bird, a partner group product manager at Microsoft who leads responsible AI for foundational technologies.

Today at Microsoft Build, the company announced that Azure AI Content Safety is in preview. This new Azure AI service helps developers create safer online environments and communities with models that are designed to detect inappropriate content across images and text. The models assign a severity score to flagged content, indicating to human moderators what content requires urgent action.

“It’s the safety system powering GitHub Copilot, it’s part of the safety system that’s powering the new Bing. We’re now launching it as a product that third-party customers can use,” said Bird.

Azure AI Content Safety is integrated into Azure OpenAI Service, providing customers of generative AI seamless access to it. The service can also be applied to non-AI systems, such as online communities and gaming platforms, and the filters can be fine-tuned for context. For example, the phrase, “run over the hill and attack” used in a game would be considered a medium level of violence and blocked if the gaming system was configured to block medium severity content. An adjustment to accept medium levels of violence would enable the model to tolerate the phrase, Bird explained.

In addition, Microsoft announced new media provenance capabilities coming to Microsoft Designer and Bing Image Creator that will enable users to verify whether an image or video was generated by AI. The technology uses cryptographic methods to mark and sign AI-generated content with metadata about its origin.

A new platform for development

About four years ago, Scott and his colleagues hypothesized that generative AI systems based on large language models would become platforms. They would allow developers to build new applications and services with relatively little effort compared to more classical forms of machine learning that require getting data, choosing a model, training it, testing it, deploying it and so on.

“The point where we are today is just fantastic. You can take a large language model like GPT-4 and start using that to build applications,” Scott said. “We’ve established this new application platform called copilot.”

Microsoft believes copilot represents both a new paradigm in AI-powered software and a profound shift in the way software is built – from imagining new product scenarios, to the user experience, the architecture, the services that it uses, and how to think about safety and security. The Microsoft ecosystem, Scott added, is the place for developers to build copilots end-to-end.

“We have everything you need on Azure for making a copilot,” he said. “And those things work super well together, so trying your idea and iterating quickly will be easier to do on top of Azure than it will be any other way.”

Related

• Learn more about Bing, Microsoft 365 Copilot, Dynamics 365 Copilot, Copilot in Microsoft Viva and Microsoft Security Copilot
• Read: How 6 Experts Use Next-Generation AI
• Read: Reflecting on our responsible AI program: Three critical elements for progress
• Read: How Microsoft’s bet on Azure unlocked an AI revolution

Top image: Microsoft is introducing tools that will help developers build their own copilots and next-generation AI apps. Graphic courtesy of Microsoft.

John Roach writes about research and innovation. Connect with him on LinkedIn.