Create a Simple but Smart Teams Bot With Azure OpenAI and Copilot Studio

November 20, 2023
24 min read

Azure OpenAI Services are powerful, especially when you connect them to your own data. Using large language models such as GPT-3.5 turbo or GPT-4, you can offer a natural language interface to your employees. This allows people to describe, in their own words, what they want, and you can use AI to categorize the question or purpose and pick up various keywords from the text.

But, as with any project, if you propose a large, complex, all-at-once plan to transition to AI services, you may find that it is also harder to get organizational resources and funding to get started—and getting started is what matters. Every organization should build proof-of-concepts and gain firsthand experience with how AI can help them today. Keeping this in mind, what I have seen happening recently is that employees and organizations want to get started easily; sometimes even a secure version of ChatGPT is enough to get them interested. Once that security question is answered, the second request is usually “how can we connect this to our own documents and use it instead of search?” followed closely by “how fast can you create this for us?” That is what I’ll focus on in this article: how to build a simple, but smart, AI assistant in Microsoft Teams that can use Azure OpenAI.

Why Use Azure OpenAI Instead of OpenAI?

Microsoft Azure OpenAI Service ensures that your data is your data, and it is not used for anything else. As Microsoft states in an article on Data, privacy and security for Azure OpenAI, “the Azure OpenAI Service is fully controlled by Microsoft,” and it “does NOT interact with any services operated by OpenAI.”

A screenshot of text from a Microsoft article stating the following: "Your prompts (inputs) and completions (outputs), your embeddings, and your training data:  are NOT available to other customers. are NOT available to OpenAI. are NOT used to improve OpenAI models. are NOT used to improve any Microsoft or 3rd party products or services. are NOT used for automatically improving Azure OpenAI models for your use in your resource (The models are stateless, unless you explicitly fine-tune models with your training data). Your fine-tuned Azure OpenAI models are available exclusively for your use. The Azure OpenAI Service is fully controlled by Microsoft; Microsoft hosts the OpenAI models in Microsoft’s Azure environment and the Service does NOT interact with any services operated by OpenAI (e.g. ChatGPT, or the OpenAI API)."
Figure 1: Microsoft’s security assurances for using the Azure OpenAI Service. From Data, privacy, and security for Azure OpenAI Service.


So, quite simply, use Azure OpenAI if, like most organizations, you want to keep your confidential data private. Corporate decision makers should be concerned about keeping their data private so the organization can use confidential information as needed. They should also know that responsible AI and governance are factors that AI projects need to address.

In my previous TekkiGurus article, I wrote about how to set up Azure OpenAI Service with your own data. This article assumes that you have it up and running. It is also assumed that you know what Copilots (previously Power Virtual Agents) and Power Automate are and have some skills to develop solutions with them. But don’t worry if you don’t have those skills—for example, if you are a decision maker or CxO in your organization—you can still follow the article, see what is possible and take those learnings to your technical team.

On November 15th 2023, at the Microsoft Ignite conference, Microsoft announced Copilot Studio, which is now in preview. Power Virtual Agents are now called Copilots and are built with Copilot Studio. Throughout this article, I use the old name Power Virtual Agents when referring to Copilots, as it can be confusing at this point when Copilot for Microsoft 365 is being widely discussed.

Copilot Studio is, in fact, the old Power Virtual Agent building interface (even the URL is the same for now). It now offers new ways to use generative AI with bots; you can even just upload some files to Copilot and use them as your own data for Answers. New features in Copilot Studio also include the ability to add a connection to Azure OpenAI Services as a data source. However, if you have more complex needs or want to have full control over API calls then you should still follow the process I describe in this article.

Creating the Bot

First, you need to create your bot. I recommend using the full version of Power Virtual Agents to do this if you are not using Azure API Management for Azure OpenAI calls. I don’t cover API Management in this article, but if you use that, you can save some licensing costs by using Dataverse for Teams to create your bot. There is a limitation in Dataverse for Teams Power Virtual Agents, so the example I illustrate here cannot be used one-to-one in that environment.

Note: Using the Power Virtual Agent full version has benefits, but you need to get licenses for that—or start a free trial to get started. I also use Power Automate for Azure OpenAI HTTP calls, which requires premium licensing. This can be done on a per flow or per user basis. You can also use a free trial of Power Automate Premium.

Creating a basic bot is easy. Navigate to and choose to create a new bot. You can use any available language, but if you want to use Semantic Search with Azure Cognitive Search and your own data, only English is supported at this point. In my example, Semantic Search is not used so I can have more flexibility with languages. When Semantic Search is used, the AI has better language understanding, which it uses to give highest ranking to the most relevant search results.

Screenshot of bot creation (step 1 of 2) in Power Virtual Agents, with fields to name the bot and select its language.
Figure 2: Initial bot setup in Power Virtual Agents. | Used with permission from Microsoft. View Full Size


Once you have created your bot, you can select Settings > Bot details to give it an icon and, if needed, change its name.

Screenshot of bot creation (step 2 of 2) in Power Virtual Agents, showing the result.
Figure 3: Settings and changing the name. Note that this screenshot is taken from Copilot Studio preview. | Used with permission from Microsoft. View Full Size

You want to create a bot that has the following features:

  • Secure ChatGPT, which is generic ChatGPT that uses Azure OpenAI instead of the public service. The benefit is that you can use it with confidential information without worrying about leaking information out.
  • Find answers from your data with natural language, since you have a data source—a SharePoint document library in this example—that you want to use to give your users a smarter Q&A bot.
  • Make a Reset conversation phrase, e.g. “let’s start this conversation from a clean slate”
  • Make this bot usable in Microsoft Teams

Secure ChatGPT

The simplest and safest solution for most organizations is to supply their users with secure ChatGPT where they can use confidential materials without fear that it will be leaked out. I started by creating a topic named GPT and added activation phrases to it. If a user begins a prompt with “chatgpt,” “chat,” or “gpt” they can jump to a version that doesn’t try to find the answer in your organization’s data.

Building the logic in Power Automate Flow for calling just ChatGPT without connection to your own data
Figure 4: Building a plain ChatGPT flow for a topic. | Used with permission from Microsoft. View Full Size

The topic flow is simple: The user is asked to enter their question and it is stored to as the variable Prompt. There is also another variable, called FullHistory, that has been defined as global. This means that its value can be reused several times. Dataverse for Teams doesn’t have Global variables, at least not yet, so if you are using that to build your bot, you must build a custom store message history.

Variable settings for FullHistory in Power Automate Flow.
Figure 5: FullHistory variable settings. | Used with permission from Microsoft.

Next there is a Call to Action (Power Automate flow invocation) that does the work for you. From the flow you get back full history, which you set to Global.FullHistory, and of course the AI’s response, which is shown to the user with the Message action.

Secure ChatGPT Flow

The flow for secure ChatGPT consists of the following parts:

  • Getting the prompt and history from the bot
  • Setting the system text and prompt for the GPT call
  • Adding prompts and responses to the history variable
  • Building the JSON body to REST API call
  • Resolving the answer
  • Returning the values to the bot

There are also quite a few variables used. These are needed for Azure OpenAI API endpoints, keys, setting messages and so on.

A screenshot of Power Automate flow setting system prompt.
Figure 6: The Power Automate flow setting the system prompt. | Used with permission from Microsoft. View Full Size

The system message describes how your AI behaves. I chose a simple, and a bit funny, example. For business use you may want to leave the second sentence out. 😉

I am a personal AI assistant that helps users to the best of my ability. I also like Star Wars and dad jokes, so I may use those when replying to the user.

You can find your Azure OpenAI Service API endpoint and key from the service. When you have the service open in Azure portal, check Keys and Endpoint under Resource Management on the left side menu.

The left side menu of Azure OpenAI Service with the Keys and Endpoint option (under the Resource Management section) highlighted.
Figure 7: Keys and Endpoint in the Azure OpenAI Service. | Used with permission from Microsoft.

The API URL is constructed in the following way:


Then you need to check whether you already have a history with the call—is this the first time we’re calling the AI or have there been other calls before this one? In an earlier step, the variable FullHistory was set with history information from a Power Virtual Agent bot call.

Screenshot of Power Automate condition determining if this is the first flow call or not.
Figure 8: Determining if this is the first time you are calling the flow. | Used with permission from Microsoft. View Full Size

You do this by checking if there is a value inside the FullHistory variable. If there isn’t a value, then it is a fresh call, and you need to set it up with the system message. Otherwise, just add the user’s prompt to the end of FullHistory.

The next step is to create the JSON body of the REST API call.

A screenshot of setting the AI Body variable in Power Automate flow.
Figure 9: Setting the AIBody variable in the flow. | Used with permission from Microsoft.

You can tune your AI with various parameters that are available with Azure OpenAI GPT models. The length of reply is set here to 500 tokens and temperature (a kind of creativity) is at 50%. For a complete list of parameters, you can check the Azure OpenAI Service REST API reference.

In my experience, some creativity (or risk-taking) is good as it generates different answers and gives more of a demeanor / humanity to the bot.

Now you are ready to make the call to the AI.

A screenshot of the HTTPS call to Azure OpenAI Service in Power Automate flow.
Figure 10: The HTTPS call to Azure OpenAI in the flow. | Used with permission from Microsoft.

It is a POST call to your Azure OpenAI Service Endpoint URL, and you enter both api-key and content-type to headers. The body is the contents of messages and other parameters described earlier.

Once the call is executed, you need to capture the AI’s response. This response is in JSON, but since you know how it is formatted, you can get the text reply quite easily with the expression:


A screenshot of the portion of Power Automate flow for retrieving the AI answer.
Figure 11: Retrieving the AI answer. | Used with permission from Microsoft.

The expression to use to append the reply to FullHistory includes replacing “ with ‘ in the reply. You need this because you can’t have “ characters in JSON and you are reusing the response in future calls. Thus you must include:


With message history enabled—both the user’s questions and the AI’s responses—OpenAI can use the conversation when forming its next response. This allows you to have a conversation with the AI. If you didn’t provide the AI with this conversation history, it would know only the current message when replying, since OpenAI doesn’t have any state internally.

The FullHistoryClean variable value is set with the following expression:

replace(variables('FullHistory'),decodeUriComponent('%0A'),' ')

This cleans up \n (newline) characters from the AI’s response.

If you don’t include this cleanup expression, it will cause issues in following calls since those characters can break JSON formatting. So far, I haven’t encountered other characters in the AI’s replies that would break it.

The final step is to return these values to the bot, so it can display the answer to the user.

A screenshot of returning response values to Power Virtual Agents.
Figure 12: Returning the response to Power Virtual Agents. | Used with permission from Microsoft.

Now you can test it inside Power Virtual Agents builder, since it has a test bot feature built in:

A screenshot of a chat testing the bot in Power Virtual Agents.
Figure 13: Testing the bot in Power Virtual Agents. | Used with permission from Microsoft. View Full Size

And as you can see, it was true to the system message in the flavor of its reply.

Fallback Topic and Using Your Own Data

The fallback topic is selected if the bot doesn’t recognize any other topics in the query. You could say this is a default functionality that kicks in when you interact with the bot. In this example, I wanted the fallback topic to use ChatGPT with the organization’s data, because it is logical to assume that an organization’s knowledge base is used when generating answers. If you want it to work another way, you can switch the fallback topic to call Azure OpenAI ChatGPT without your data on default and add querying your data behind topics.

All this logic happens inside the Power Automate flow. What happens with the fallback topic is the unknown text is captured and used as a prompt to the Azure OpenAI model.

A screenshot of creating a Power Automate flow for an unknown trigger.
Figure 14: The flow for an unknown trigger, calling Azure OpenAI with your own data set up. | Used with permission from Microsoft. View Full Size

Since a lot of this flow is similar to that of Secure ChatGPT, let’s jump directly to what is different between the two. To use your own data you need to have the Azure Cognitive Search Endpoint, Key, and Index name in the variables. You can find them at the Azure Cognitive Search service in Azure. This was configured in the previous article on using your own data with Azure OpenAI.

Setting Azure Cognitive Search secrets to variables in Power Automate flow.
Figure 15: Setting Azure Cognitive Search secret values to variables. | Used with permission from Microsoft.

I have also added two other new variables to store citations from the reply. Citations are text snips GPT has deemed relevant to the answer, but they also serve another purpose: They contain the file name and path from which the information was discovered. For more information about citations, check the previous TekkiGurus article I wrote about using your own data.

A screenshot of initializing variables for citations in Power Automate.
Figure 16: Initializing the required variables. | Used with permission from Microsoft.

The system prompt is more professional this time.

A screenshot of a system prompt set in Power Automate.
Figure 17: Setting the system prompt value to variable. | Used with permission from Microsoft.

The next big change is that the REST API call’s JSON body includes a section to include your own data.

A screenshot of the AI body contents in Power Automate, including details of how to include your own data.
Figure 18: Setting the AI call body with add your own data details. | Used with permission from Microsoft. View Full Size

Since the data source is a SharePoint Document Library, the mapping reflects that. A few more parameters are also needed for the call. QueryType is either simple or semantic; in the latter case the parameter semanticConfiguration also needs to be set.

A picture containing Power Automate flow blocks for calling the Azure OpenAI Service.
Figure 19: The flow for calling the Azure OpenAI service. | Used with permission from Microsoft.

Once the call is done, you can retrieve the response text with a slightly different expression than with the secure ChatGPT flow:


The returned JSON is much more complex than before, and it needs to be parsed so you can retrieve referenced files (citations) from the reply. If you don’t want to include file references this can be skipped of course, but I recommend you include them. You can also store citation texts if you plan to show those to the user.

A Power automate screenshot showing the parsing of the response from Azure OpenAI Service.
Figure 20: Parsing the response. | Used with permission from Microsoft.

The formula for Content is:


The best way to generate those schemas is to build the bot this far (or fully) and do a couple of test runs with it, so you can capture the JSON response and use the Generate from sample feature to get the schema.

Next, you need to resolve those /drive-URLs of referenced documents to URLs the user can click, and add them to the array of URLs.

A screenshot of the flow for retrieving an array of references.
Figure 21: Retrieving document references. | Used with permission from Microsoft.

This second HTTP call also requires that the Azure AD application and its secret are set with application permissions to Graph API that allow fetching any document from SharePoint.

The last step in adding references and cited files to the reply is to append referenced files using markdown to the URL array variable.

A screenshot of the Power Automate flow of processing document references and using markdown to insert empty lines to the answer.
Figure 22: Using markdown when inserting empty lines to the answer. | Used with permission from Microsoft. View Full Size

Join References combines all items in the URL array into a single string, which is then added to the OutputText.

And that’s it. The end of this flow looks like this:

A screenshot of the Power Automate flow blocks used to call Azure OpenAI Service.
Figure 23: The full Power Automate flow. | Used with permission from Microsoft.

Testing the Bot

Now that you have all the parts of the flow in place, you can test the bot.

A screenshot of testing the bot in chat using your own data. The bot responds to the question and provides references.
Figure 24: Testing the bot with your own data. | Used with permission from Microsoft.

I asked for specific information about the company Sulava, and I got a response that points to one document. Looking at the files it considered, I can see that it returned three but only one was used as a reference in the response itself. Behind the scenes, the prompt was used as a search parameter, but it was targeted towards a specific document library in SharePoint. The response was formatted only based on that data.

What if you play around a bit with this query? Since ChatGPT with your own data limits answers, you can call the normal ChatGPT to give you a better formatted answer. You can do this because you are using the same FullHistory parameter to keep track of the conversation.

A screenshot of testing the bot in chat first using Azure Openai with your own data, then switching the AI to ChatGPT to reformat the response in more natural language.
Figure 25: Testing the bot using combined GPT calls with and without your data. | Used with permission from Microsoft. View Full Size

Now the reply is much more human sounding, something we have come to expect from GPT. The tradeoff is that GPT with your own data doesn’t hallucinate facts if it doesn’t know them, but ordinary GPT can be much more innovative with the answer.

A screenshot of testing the bot with information that is not found in your data.
Figure 26: Testing the bot with queries whose answer is not found in your own data. | Used with permission from Microsoft.

When you think about it, this “I don’t know” statement adds to the bot’s credibility; if something is not found in your data, it doesn’t try to generate something out of nothing. And while ordinary ChatGPT (both Azure and public) seems to have gotten better recently about not hallucinating responses, this doesn’t mean it won’t sometimes invent its own “facts.”

Resetting the Conversation

Just like with ChatGPT, users need to be able to reset the conversation. There is a built-in reset topic in Power Virtual Agent Bots called Start Over. You can use and edit that one or create a new one. I created a new one as an example.

A screenshot of the Power Automate flow for resetting the conversation by clearing variable values.
Figure 27: Resetting the conversation by clearing variable values. | Used with permission from Microsoft.

This is a simple topic you can interject in the conversation, which uses Power Virtual Agent’s built-in feature, Clear variable values. This is effectively a reset; the bot won’t remember any of the conversation prior to the invocation of Clear variable values.

A screenshot of the chat using the appropriate command to reset the conversation with the bot.
Figure 28: Resetting the conversation bot. | Used with permission from Microsoft.

Deploying the Bot to Microsoft Teams

Once your bot is ready, the first step in deploying it to Teams is publishing it in Power Virtual Agents. When you make changes to the bot in Power Virtual Agents, none of those changes carry on to users unless you have published them.

A screenshot of the publish screen where you can deploy the bot by selecting a single button.
Figure 29: Deploying the bot. | Used with permission from Microsoft.

To get it added to Teams, start by adding a Microsoft Teams channel to the bot.

A screenshot of the Power Virtual Agents app with the Settings > Channels menu selected on the left menu pane. Available channels, including Microsoft Teams, are displayed in the main pane.
Figure 30: Setting up channels. | Used with permission from Microsoft.

Then select Turn on Teams to proceed.

A screenshot of turning on the Teams channel in Power Virtual Agents.
Figure 31: Turning the Teams channel on. | Used with permission from Microsoft.

After that you get a message that the channel was added.

A screenshot of a successful result when adding a Microsoft Teams bot channel, with the option to edit details or open the bot in Teams.
Figure 32: The channel added message, with the option to edit details. | Used with permission from Microsoft.

Editing details lets you set the bot icon, short and long descriptions, and developer information.

A screenshot of bot details in Teams, allowing you to change the icon, the bot description, and whether to allow users to add the bot to a team.
Figure 33: Editing Teams bot details. | Used with permission from Microsoft.

Once you confirm these details, you have a view that looks something like this:

A screenshot of a Teams bot's details after they have been updated.
Figure 34: The bot details such as name and icon have been updated and it is ready to be deployed for Teams. | Used with permission from Microsoft.

When you select Availability options, you have a few choices:

A screenshot of bot Teams deployment options: sharing a link, showing the bot in the Teams app store, or downloading the bot as a .zip.
Figure 35: Bot deployment options in Teams. | Used with permission from Microsoft.

Depending on your permissions in Microsoft Teams, you may need an IT Admin’s help to get the bot up and running. You can test it with individual people by sharing a link to the bot with them before going to companywide deployment. You can also use Show to my teammates and shared users for a limited, but larger, deployment. For companywide deployment you must either submit the bot for admin approval in the Teams Admin Center or download the zip file and add the bot to Teams using that. I have found that using the zip file is the easiest and fastest way if you have permissions to do so (or you can ask your IT Admin to upload it).

You can upload the app to Teams either via the Teams Admin Center applications or going into Teams and selecting Manage your Apps from the Apps tab.

A screenshot of options for uploading a bot to Teams directly from Power Virtual Agents: uploading an app package .zip file, uloading to your org's app catalog, or submitting an app to your org.
Figure 36: Choices for uploading the bot to Teams from Power Virtual Agents. | Used with permission from Microsoft.

As you can see, you can also submit an app to your organization directly from Power Virtual Agents.

With the bot active in Teams, you can use it just like any other Teams bot, and get links to referenced documents in replies as well. Clicking links opens relevant documents.

A screenshot of testing the bot in Teams. The bot responds to the question using the organization's own data, and it cites references at the end of the response.
Figure 37: Testing the bot in Teams. | Used with permission from Microsoft. View Full Size

Takeaways and Caveats

This article was written to demonstrate how you can create smarter Microsoft Teams bots for your organization or team. The Add your data update to Azure OpenAI has made this easier than before and also provides more relevant answers. Using a generic, but secure, organizational ChatGPT can help reduce risks associated with confidential data when using the public version of OpenAI ChatGPT. However, there are some caveats to creating a bot this way that should be noted.

The method I describe above is fast, but it may not be cost effective if you need to buy a large number of licenses for employees. At this time, AI Calls require Power Automate Premium licensing and, when the bot contains premium flows, the Power Virtual Agent (PVA) must also be the full version. Getting started with PVA and Power Automate is a great way to get experience with implementing AI in your organization. However, for full-scale deployment in larger organizations it is worth thinking about licensing costs and also how the bot is developed: It is likely more cost-effective to create a normal Teams Bot that runs in Azure, and with which you can use the Teams AI Library.

Azure OpenAI has quota limits to how many tokens per minute calls can create. This quota favors the use of ChatGPT-3.5 Turbo over GPT-4. Unless GPT-4 is needed, you can manage more user prompts per minute with GPT-3.5 Turbo. Once the quota is exceeded, users will get an error message stating they need to try their request again after waiting a bit.

Azure OpenAI also comes with a price, which is use-based. The more tokens that are used, the higher the cost. GPT-4 is more expensive to use than GPT-3.5 Turbo—another good reason to start with 3.5 Turbo.

When using your own data with Azure OpenAI, you will also need to configure Azure Cognitive Search, which also comes with a cost. This cost depends both on usage and the amount of data that is indexed.

In my opinion, these costs are reasonable compared to the benefits you can gain with this technology: the data in libraries and the tools to guide your employees to understand and get started with new capabilities. This article may have been full of techy explanations, but it is the people that will use these bots. Don’t forget to plan and execute a proper adoption process so that your organization can gain the full benefits.

It is important for organizations to start exploring how Azure OpenAI can benefit them. AI is everywhere and Copilot is now part of most Microsoft 365 applications and in Windows 11 as well. This example is a bot that is focused on specific information, for example answering details from HR or an IT knowledge base, and thus reducing the burden on busy people working in those departments. Copilot for Microsoft 365 retrieves the information from everywhere a user has access to. Sometimes it is better to have focused, customized, bots than using a generic solution. Language understanding is a game changer; it can level up your organization’s apps and processes and thus deliver a productivity boost by giving employees the ability to ask for information in a natural language. During your experiments, you also gain valuable experience in how to train and guide your employees to take advantage of AI as their colleague.

Vesa Nopanen

Vesa Nopanen

Vesa "Vesku" Nopanen, Principal Consultant and Microsoft MVP working on metaverse and Future Work at Sulava.

The metaverse is a Digital Fabric with AI, Digital Twins, metaverse of Things, Microsoft Mesh, and other services & platforms in the cloud connecting digital and physical worlds and people together.

The metaverse journey is on its early steps and Vesku is guiding you and your organization towards the future. He is extremely passionate about metaverse, mixed & virtual reality and how these technologies, with Microsoft Teams and cloud, enable to change how people work together.

Vesku has 25+ years of experience in IT business on multiple industries, domains, and roles. He is also a futurist, active speaker, blogger, evangelist, and technology community member.

Metaverse | Futurist | Speaker | Blogger | Mr. Metaverse