Microsoft outlines framework for building AI apps and copilots; expands AI plugin ecosystem - Source (2023)

Remember when software wasn’t connected to the internet? Didn’t think so. In a handful of years, this will also be true for software that doesn’t come with an intelligent copilot assistant, according to Kevin Scott, Microsoft’s chief technology officer.

A copilot is an application that uses modern AI and large language models to assist you with a complex cognitive task – from writing a sales pitch or catching up on a missed meeting to generating images for a presentation or planning a themed dinner party.

Microsoft introduced the concept of a copilot nearly two years ago with GitHub Copilot, an AI pair programmer that assists developers with writing code. This year, Microsoft rolled out copilot experiences across its core products and services, from the AI-powered chat in Bing that’s changing how people search the internet to Microsoft 365 Copilot, GitHub Copilot X, Dynamics 365 Copilot, Copilot in Microsoft Viva and Microsoft Security Copilot.

Today at the annual Microsoft Build developers conference, Microsoft announced that it has expanded this ecosystem of Microsoft Copilots to include Copilot in Power BI and Copilot in Power Pages in preview, Copilot in Microsoft Fabric, available in preview soon, and Windows Copilot, which will start to become available for preview in June. The company also introduced new features that will help developers build their own copilots and next-generation AI applications. This includes new tools called plugins that make copilots more useful by allowing them to interact with other software and services.

“You may look at Bing Chat and think this is some super magical complicated thing, but Microsoft is giving developers everything they need to get started to go build a copilot of their own,” Scott said. “I think over the coming years, this will become an expectation for how all software works.”

Expanding the plugin ecosystem

OpenAI introduced ChatGPT plugins in March. Microsoft announced plugins for Bing earlier this month. That technology allows ChatGPT and Bing Chat to help you find and book a restaurant reservation using an OpenTable plugin, for example.

Today, Microsoft is adopting the same open plugin standard that OpenAI introduced for ChatGPT, enabling interoperability across ChatGPT and the breadth of Microsoft’s copilot offerings. That means developers can now use one platform to build plugins that work across both business and consumer surfaces, including ChatGPT, Bing, Dynamics 365 Copilot, Microsoft 365 Copilot and Windows Copilot. Microsoft also announced it is bringing Bing to ChatGPT as the default search experience.

(Video) Accelerate development with Visual Studio and Microsoft Power Platform | BRK272H

As part of this shared plugin platform, Bing is adding to its support for plugins. In addition to the ones previously announced for OpenTable and Wolfram Alpha, it will also have Expedia, Instacart, Kayak, Klarna, Redfin, TripAdvisor and Zillow among many others in the Bing ecosystem.

In addition, developers will now be able to extend Microsoft 365 Copilot with plugins. Plugins for Microsoft 365 include ChatGPT and Bing plugins, as well as Teams message extensions and Power Platform connectors – enabling developers to leverage their existing investments. And developers will be able to easily build new plugins with the Microsoft Teams Toolkit for Visual Studio Code and Visual Studio.

How plugins work

At Microsoft Build, the company showcased an AI development framework that helps developers build their own copilot. This stack includes support for plugins that augment the capabilities of AI systems by allowing them to retrieve real-time information, incorporate company or other business data, perform new types of computations and safely take action on the user’s behalf.

At root, think of plugins as a bridge. This could be a bridge between a large language model that was trained on public data from the internet and all the data that a company may keep privately about its benefits. The plugin is the bridge that gives the copilot access to those files when it answers a question from an employee at the company.

Similarly, a plugin could serve as a bridge between a large language model and a website or back-end system that a company uses to book business travel, enabling the copilot to make arrangements for a salesperson’s trip to San Francisco that are in line with the company’s travel policy.

Microsoft outlines framework for building AI apps and copilots; expands AI plugin ecosystem - Source (1)

“A plugin is about how you, the copilot developer, give your copilot or an AI system the ability to have capabilities that it’s not manifesting right now and to connect it to data and connect it to systems that you’re building,” Scott said. “I think there’s going to eventually be an incredibly rich ecosystem of plugins.”

Expanding the plugin ecosystem

Developers will also be able to create, test and deploy their own plugins in a number of ways – to eventually deploy for use with Microsoft Copilots and to augment the capabilities of their own applications built with generative AI technology.

For example, a developer at a large corporation may want the Microsoft 365 Copilot to be able to access the company’s contracts with customers and vendors in order to ask questions about how certain legal issues were handled in the past. To do that, the developer needs to connect the Microsoft 365 Copilot with the private database of legal files, which requires the documents to be encoded, indexed and stored in such a way that they can be searched.

“That’s a pattern of how you would load information into the model,” said John Montgomery, Microsoft corporate vice president of program management for the AI platform. “And that’s where plugins come in. What a plugin does is it says ‘Hey, we want to make that pattern reusable and set some boundaries about how it gets used.’”

Microsoft is releasing a set of capabilities to facilitate the creation of plugins that work across its copilot surfaces. Visual Studio Code, GitHub Copilot and GitHub Codespaces will make it simple for developers to create, debug and deploy new plugins, for example, and Azure AI will add capabilities to run and test plugins on private enterprise data. Once created, these plugins will work across Microsoft’s Copilot experiences.

(Video) ChatGPT & The Future of Everything (2023 April)

Montgomery noted that the addition of plugins will accelerate the pace at which Microsoft customers can develop innovative applications with Azure OpenAI Service, which provides access to OpenAI’s generative AI models including GPT-4 and is currently serving more than 4,500 companies.

“What you’re going to see at Build is our current best thinking about the architectural practices, the tooling you need, and the app patterns and user experience designs you want to be putting around these new copilot applications,” he said in an interview prior to the conference.

New tools for building your own copilot

While Microsoft has deployed generative AI technology into the company’s products and services, developers will eventually build most of the copilots in the world, according to Scott.

“They will understand a particular thing that they or their users are trying to accomplish, and they will use this AI software development pattern to go build those things for those users,” he said.

Copilot development begins like any software project: as an idea. What’s different is the realm of possibilities that this AI development platform enables, Scott said.

He likens this to the new experiences and services that appeared with the arrival of smartphones. Many things that were impossible to imagine became possible. Some were straightforward to develop, such as apps for familiar, casual games. Others were hard and harnessed the full capabilities of the smartphone platform to create new types of experiences, including services that disrupted the taxi and vacation rental industries.

(Video) Nvidia AI Computer, GitHub Copilot X & Adobe Firefly AI are all LIFE CHANGING

“They went from impossible to hard and because they were hard and really used the capabilities of the new platform, they found enduring success,” said Scott, alluding to his expectation that similarly disruptive applications will be built with the copilot stack.

Microsoft outlines framework for building AI apps and copilots; expands AI plugin ecosystem - Source (2)

With a solid idea for an application in hand, developers can go about turning it into reality, he said. The process starts with selecting a generative AI model such as GPT-4, which is available through Azure OpenAI Service. The next step is to give the copilot a meta-prompt, which is a base description of the copilot’s role and how it should operate. In addition, the copilot needs memory to keep track of a conversation with a user and respond with appropriate context and awareness. Finally, the copilot needs the ability to access data and other services, which plugins can provide.

Copilots deployed to support complex tasks, such as writing a whole book, will also require orchestration – the ability to break down tasks into multistep components and execute the tasks according to a generated plan.

Developers can use tools such as the Semantic Kernel software development kit that Microsoft released to open source in March to integrate large language models with conventional programming languages. The kit includes tools for memory and orchestration and support for plugins, which accelerates the development of copilots, according to Montgomery.

At Microsoft Build, the company also introduced new tools to simplify the development of copilots. For example, with the new Azure AI Studio, developers can now more easily ground conversational AI models on their private data. With a new capability called Azure OpenAI Service on your data, developers can uncover organization-specific insights from data, text and images using natural language-based application interfaces. To further extend the capabilities of large language models, Microsoft also announced support for plugins with Azure OpenAI Service. Azure AI also now supports Azure Machine Learning prompt flow, a sophisticated prompt engineering tool, noted Montgomery.

“We’ve built a bunch of these copilots. We’ve looked at a bunch of the different architectures out there for how you load other information in, how you create reproducible content, how you feed the model the right stuff so that it gives the right answers,” he said. “Build is going to be a celebration of all the things we’ve learned, the tooling we’ve created and bring that out to the world.”

(Video) #134 - The one with SAP Business One (Moshe Nachman & Richard Duffy) | SAP on Azure Video Podcast

Responsible AI updates

Importantly, developers also need to ensure the copilot returns the intended results and avoids outputs that are biased, sexist, racist, hateful, violent or prompt self-harm, noted Sarah Bird, a partner group product manager at Microsoft who leads responsible AI for foundational technologies.

Today at Microsoft Build, the company announced that Azure AI Content Safety is in preview. This new Azure AI service helps developers create safer online environments and communities with models that are designed to detect inappropriate content across images and text. The models assign a severity score to flagged content, indicating to human moderators what content requires urgent action.

“It’s the safety system powering GitHub Copilot, it’s part of the safety system that’s powering the new Bing. We’re now launching it as a product that third-party customers can use,” said Bird.

Azure AI Content Safety is integrated into Azure OpenAI Service, providing customers of generative AI seamless access to it. The service can also be applied to non-AI systems, such as online communities and gaming platforms, and the filters can be fine-tuned for context. For example, the phrase, “run over the hill and attack” used in a game would be considered a medium level of violence and blocked if the gaming system was configured to block medium severity content. An adjustment to accept medium levels of violence would enable the model to tolerate the phrase, Bird explained.

In addition, Microsoft announced new media provenance capabilities coming to Microsoft Designer and Bing Image Creator that will enable users to verify whether an image or video was generated by AI. The technology uses cryptographic methods to mark and sign AI-generated content with metadata about its origin.

A new platform for development

About four years ago, Scott and his colleagues hypothesized that generative AI systems based on large language models would become platforms. They would allow developers to build new applications and services with relatively little effort compared to more classical forms of machine learning that require getting data, choosing a model, training it, testing it, deploying it and so on.

“The point where we are today is just fantastic. You can take a large language model like GPT-4 and start using that to build applications,” Scott said. “We’ve established this new application platform called copilot.”

Microsoft believes copilot represents both a new paradigm in AI-powered software and a profound shift in the way software is built – from imagining new product scenarios, to the user experience, the architecture, the services that it uses, and how to think about safety and security. The Microsoft ecosystem, Scott added, is the place for developers to build copilots end-to-end.

“We have everything you need on Azure for making a copilot,” he said. “And those things work super well together, so trying your idea and iterating quickly will be easier to do on top of Azure than it will be any other way.”


(Video) Building the Future: How Can Architects Harness the Power of AI

  • Learn more about Bing, Microsoft 365 Copilot, Dynamics 365 Copilot, Copilot in Microsoft Viva and Microsoft Security Copilot
  • Read: How 6 Experts Use Next-Generation AI
  • Read: Reflecting on our responsible AI program: Three critical elements for progress
  • Read: How Microsoft’s bet on Azure unlocked an AI revolution

Top image: Microsoft is introducing tools that will help developers build their own copilots and next-generation AI apps. Graphic courtesy of Microsoft.

John Roach writes about research and innovation. Connect with him on LinkedIn.


Does Microsoft have an AI program? ›

Microsoft 365: AI to simplify work and life

Across Microsoft 365, AI powers innovative apps that can help you write and design better, visualize maps and charts in Excel, and streamline your inbox. Learn how AI makes Office apps easier to use, more collaborative, and more secure.

What is Microsoft's approach to AI? ›

Microsoft is committed to a practice of responsible AI by design and guided by a core set of principles: fairness, reliability and safety, privacy and security, inclusiveness, transparency and accountability.

Is Microsoft AI course free? ›

If you're looking for a free introductory AI course for beginner's, Microsoft has got you covered.

Is Microsoft Copilot free? ›

Microsoft sells GitHub Copilot, its developer-focused AI pair-programming technology, for US$10 per user per month (or US$100 per user per year) for individuals. GitHub Copilot plugs directly into developer's' editors. And GitHub Copilot for Business is US$19 per user per month.

What is the name of Microsoft AI platform? ›

Azure AI Platform—Artificial Intelligence. Microsoft Azure.

What are three Microsoft guiding principles for responsible AI? ›

Microsoft AI guiding principles

At Microsoft, we've recognized six principles that we believe should guide AI development and use: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability.

What AI does Microsoft own? ›

Microsoft announced a new multiyear, multibillion-dollar investment with the artificial intelligence lab OpenAI. The investment is the third phase of the partnership, following Microsoft's previous investments in 2019 and 2021.

How much does an AI program cost? ›

Nvidia makes most of the GPUs for the AI industry, and its primary data center workhorse chip costs $10,000. Analysts and technologists estimate that the critical process of training a large language model such as GPT-3 could cost over $4 million.

Is Microsoft AI-900 certification worth it? ›

Using the AI-900 to learn skills

The AI-900 exam will not teach you any skills. However, if you're looking to gain a foundational level of understanding about artificial intelligence and machine learning in general and Azure in particular, preparing for the AI-900 test will increase your knowledge base.

What is the salary of AI developer in Microsoft? ›

The average salary of a Artificial Intelligence Engineer at Microsoft Corporation is ₹ 30.3 Lakhs per year which is 278% more than average salary of a Artificial Intelligence Engineer in India which receives a salary of ₹ 8.0 Lakhs per year.

How much is Microsoft Copilot subscription? ›

Pricing for GitHub Copilot for Business

The GitHub Copilot for Business subscription is available on a monthly cycle, and is billed at $19 per user per month.

Is Microsoft Copilot open to public? ›

In recent news, Microsoft has released a 365 Copilot Early Access Program which grants a few users exclusive access to the new Office applications. This program is invitation-only and Microsoft has already started rolling out the first set of invites to 600 customers globally.

What is the alternative to Google Copilot? ›

Microsoft's Github Copilot Alternative – Google's Duet AI

Nevertheless, Google introduces Duet AI as an alternative tool that competes with Github Copilot. Duet AI offers real-time code generation, code review assistance, and additional capabilities like generative AI applications and text-to-image foundation models.

What is Google's AI called? ›

That is the Shakespearean question an Associated Press reporter sought to answer while testing out Google's artificially intelligent chatbot. The recently rolled-out bot dubbed Bard is the internet search giant's answer to the ChatGPT tool that Microsoft has been melding into its Bing search engine and other software.

What is Microsoft AI VS code? ›

VS Code Tools for AI is a cross-platform extension that supports deep learning frameworks including Microsoft Cognitive Toolkit (CNTK), Google TensorFlow and more. Because it's an IDE we've enabled familiar code editor features like syntax highlighting, IntelliSense (auto-completion) and text auto formatting.

How do I install Copilot on Microsoft 365? ›

Search for "GitHub Copilot" in the Extensions view. Click on the "Install" button next to the "GitHub Copilot" extension. Wait for the installation to complete.

What are the 4 key principles of AI? ›

The team at the Institute for Ethical AI and ML has assembled eight principles that can be used to guide teams to ensure that they are using AI responsibly. I'd like to run through four — human augmentation, bias evaluation, explainability, and reproducibility.

What are the 4 key principles of AI development? ›

This article provides a a breakdown of four of the eight principles: bias evaluation, explainability, human augmentation, and reproducibility. In a sense, AI models carry inherent biases as they are designed to discriminate towards the relevant answers.

Did Elon Musk own OpenAI? ›

Musk was an OpenAI cofounder and invested $50 million into the company. He left the startup in 2018. He criticized OpenAI's business model and its relationship with Microsoft, one of its current investors.

How does Microsoft seeing AI work? ›

About Seeing AI

Point your phone's camera, select a channel, and hear a description of what the AI has recognized around you. With this intelligent camera app, just hold up your phone and hear information about the world around you.

Who is Microsoft partner to build massive AI? ›

Cloud computing: Microsoft and Nvidia are building a 'massive' AI supercomputer. Here's why. The partnership makes Microsoft Azure the first public cloud to incorporate Nvidia's full AI stack -- its GPUs, networking, and AI software.

Which AI company is owned by Microsoft? ›

Microsoft announced a new multiyear, multibillion-dollar investment with the artificial intelligence lab OpenAI.

What does Microsoft AI Builder do? ›

AI Builder is a Microsoft Power Platform capability that provides AI models that are designed to optimize your business processes. AI Builder enables your business to use intelligence to automate processes and glean insights from your data in Power Apps and Power Automate.

Is there an AI like Cortana? ›

Although Weapon is an exact copy or original clone of Cortana, they aren't the same in every way. However, she plays a big role in Chief's new adventure and is a new AI to guide you on your missions. We can say that she is rather an AI created from the same DNA as Cortana.

Is Microsoft Copilot available now? ›

Last Updated on May 10, 2023

This program is invitation-only and Microsoft has already started rolling out the first set of invites to 600 customers globally. Unfortunately, there is no confirmed date for the wider release of Copilot.

What AI company has Elon Musk invested in? ›

The billionaire was named as a director in a business-incorporation document filed in March for a new company called Corp. The move comes despite Musk recently signing an open letter calling for a pause in training more powerful systems, alongside various AI researchers.

What AI company did Apple buy? ›

Apple has quietly acquired a Mountain View-based startup, WaveOne, that was developing AI algorithms for compressing video.

Does Elon Musk own an artificial intelligence company? ›

Elon Musk has launched a new AI company incorporated in Nevada as part of the billionaire's plan to create a new super company. Musk is the sole listed director of the company, which he called X.AI Corp., according to The Wall Street Journal.

How does Microsoft BOT framework work? ›

It hosts bots and makes them available to channels, such as Microsoft Teams, Facebook, or Slack. The Bot Framework Service, which is a component of the Azure Bot Service, sends information between the user's bot-connected app and the bot. Each channel can include additional information in the activities they send.

Can an AI build an app? ›

AI-generated apps for Android or iOS have become increasingly popular due to their ability to streamline tasks and enhance user experiences. Building such apps using an app builder or app maker that features AI algorithms can help creators save time and effort while achieving greater functionality and innovation.

What is Microsoft Jarvis? ›

JARVIS (Joint AI Research for Video Instances and Streams) is an AI-powered video analytics tool aimed towards enhancing security, safety & compliance. By default, Jarvis works with Azure AD.

What is the most advanced AI app? ›

Google Assistant

Without a doubt, Google is at the forefront of AI innovation and development, and its virtual assistant is considered to be one of the most advanced of its kind on the market.

What is replacing Cortana? ›

Braina. Braina is an innovative virtual assistant that provides an efficient and personalized experience to its users. It is a viable alternative to Cortana on Windows 10 and has numerous features that make it a better choice for some users.

What is the most realistic AI? ›

4 Robots That Look Like Humans
  1. Ameca the Humanoid Robot — Most Expressive. (Credit: Engineered Arts) ...
  2. Sophia the Robot — Android Ambassador. A humanoid robot named Sophia spoke at the AI for Good Global Summit in Geneva, Switzerland in 2017. ( ...
  3. Boston Dynamics Atlas — The Robot That Can Do Parkour. ...
  4. Valkyrie — The NASA Robot.
Mar 16, 2023


1. # 162 The Great ChatGPT and Translation Debate
2. First Look at ChatGPT Plugins - OpenAI's New Gamechanger
(Dave Lee on Life)
3. Microsoft 365 Platform Community Call – 26th of April, 2022
(Microsoft 365 & Power Platform Community)
4. Will ChatGPT replace Programmers? | Making a website using ChatGPT, GitHub Copilot and AI
( Code with Scaler)
5. DEMO: Secure Coding with GitHub and Copilot | Peter Murray | The DEVOPS Conference - Global 2023
6. IBC CEP: Introduction to Rust
(IBC Media)


Top Articles
Latest Posts
Article information

Author: Ouida Strosin DO

Last Updated: 09/03/2023

Views: 5847

Rating: 4.6 / 5 (76 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Ouida Strosin DO

Birthday: 1995-04-27

Address: Suite 927 930 Kilback Radial, Candidaville, TN 87795

Phone: +8561498978366

Job: Legacy Manufacturing Specialist

Hobby: Singing, Mountain biking, Water sports, Water sports, Taxidermy, Polo, Pet

Introduction: My name is Ouida Strosin DO, I am a precious, combative, spotless, modern, spotless, beautiful, precious person who loves writing and wants to share my knowledge and understanding with you.