Gosia and AI
(as Artificial Intelligence is already around us!)
Gosia and AI
Cognitive Services

Get Started with Azure Cognitive Services

Published

What are Cognitive Services?

Cognitive services are pre-built AI APIs for applications, programming languages and platforms. They are based on Deep Learning Research (Microsoft does all data preparation) and can easily help develop an intelligent application without machine learning knowledge.

In other words – they are Algorithms as a Service implemented via APIs into your application.

Why Azure Cognitive Services?

Azure Cognitive Services are easy to use and require only a few lines of code in your application to get quick results. There are also flexible — you can use your preferred programming language, such as C#, Python, Java, JavaScript, and Go. And the last — Cognitive Services are tested and built by experts from Microsoft Research, provided with quality documentation, sample code, and community support.

Cognitive Services APIs

Several categories will be changed/updated since this post comes to life. This list could be incomplete soon; you can find the full list of actual services on the Azure Cognitive Services website.

But for now (April 2023), there are five groups of Cognitive Services (some still on preview — that means you can play with them and create a proof of concept, but it is not recommended to implement those APIs on production as they could disappear for further research)

Here you can find a list of those services with a summary of what they can do for you:

Vision APIs

Computer Vision

OCR

  • OCR (Optical Character Recognition) –text recognition or text extraction.
  • Allows to extract text from images (printed or handwritten)

Image Analysis

  • Image tagging
  • Image categorisation
  • Category taxonomy
  • Image descriptions
  • Object detection
  • Brand detection
  • Face detection
  • Image type detection
  • Domain-specific content
  • Colour scheme detection
  • Adult content detection
  • Smart-cropped thumbnails

Face

Based on Responsible AI, many attributes that used to be implemented in Face API will be removed, such as emotion, age, gender, smile, makeup etc.

Access to more details Face API is available only for specific solutions that Microsoft approves.  

Spatial Analysis

  • Detect the presence and movements of people in the video.
  • Analyse social distancing
  • Count people entering the zone.
  • Face mask detection

Custom Vision

Speech APIs

Speech Services

Speech-to-text

Text-to-speech

Speech translation

Speaker Recognition

Custom keyword

Language APIs

Language Services

  • Extract information
  • Summarizer text-based content
  • Classify Text
  • Understand conversations.

Language Understanding

  • Contextual language understanding

Question Answering

  •  QnA extraction from unstructured text
  • Knowledgebase creation from collections of Q&As
  • Semantic marching for knowledge bases

Translator

  • Text Translation
  • Document translation
  • Custom translation

Decision APIs

Anomaly Detector

  • Identify potential problems early on
  • detect anomalies as they occur in real-time throughout your data set as a batch.
  • Get additional information about your data and adjust anomaly detection boundaries.

Content Moderator

  • Detected potential offensive and unwanted images.
  • Filter possible profanity and undesirable text
  • Moderate adult and racy content in videos
  • Use the built-in review tool for the best result.

Personalizer

  • Deliver rich, personalised experiences in your apps.
  • Deploy anywhere, from the cloud to the edge.
  • Understand and efficiently manage the reinforcement learning loop.

Azure OpenAI

OpenAI models are collections of Generative AI models – models that can produce new content based on what is described in the input, such as language, code and images.

Azure OpenAI have selected services from the OpenAI platform that give an environment for using them on the enterprise scale. It utilised Azure’s infrastructure, including security, compliance and regional availability.

GPT-3/GPT-4

  • Summarizing text
  • Classify text.
  • Generating names of phrases
  • Translation
  • Answering questions
  • Suggesting content

DALL-E

  • Image Generation
  • Editing an image
  • Image variation

Codex

  • Creating code suggestions

Embedded

  • A set of models that can understand and use embeddings

Other services

  • Form Recogniser
    • Extract text, key-value pairs, and tables from documents
    • Customised to your form, without manual labelling.
    • Deploy anywhere, from the cloud to the edge.

Cognitive Services Labs

Microsoft is also working on new APIs that will be available in feature to the Cognitive Services family.

All of them you can find on the Microsoft Research Site, and there is an excellent opportunity to have a real impact on those APIs by giving feedback to Microsoft on this project will be helpful and how they should evolve.

Project Academic Knowledge

Tap into the wealth of academic content in the Microsoft Academic Graph using the Academic Knowledge API.

Project Knowledge Exploration

Enable interactive search experiences over structured data via natural language inputs.

Project Conversation Learner

  • Learn from interaction examples.
  • Reduce manual coding of dialogue control logic.

Project Answer Search

  • AI that understands user queries
  • Targeted and faster results
  • Relevant results from Bing’s industry-leading web index and knowledge graph

Project Event Tracking

  • Finding events associated with Wikipedia entities.
  • List of related events organised by time.
  • Help to characterise events happening in future.

Project URL Preview

  • Quickly provides a URL’s page title, brief description, and the most relevant image.
  • Instantly enhance social scenarios, such as chat apps, to let users know what they have been sent and improve their URL-sharing experience.
  • Leverage Project URL Preview to Suppress inappropriate or adult content, block URLs on protected networks, and get users to the right content faster.

Project Gesture

Add custom hand gestures to your app.

Container Support

Cognitive Services can also be used with containers.

The benefits of using containers with Cognitive Services:

  • Control over data – allows where data are processed (it’s suitable for companies that cannot send data to the cloud)
  • Control over model updates
  • Portable architecture – the application can be created on Azure, on-premises and at the edge.
  • High throughput / Low latency – the endpoint is strictly used for billing; no customer data ever flows that way.

Services supported by containers.

Vision

Language

Speech

Decision

How to start with Cognitive Services?

Start for Free

If you don’t have an Azure account, you can sign in to get £150 credits for 30 days and 12 months to popular services + 25+ always free services.

Most Cognitive Services you can use for free for development (as those services have restrictions on the number of calls per second and number of transactions per month)

https://azure.microsoft.com/en-us/free/cognitive-services/

Azure

There are two ways to have an endpoint/key for Cognitive Services: get a bunch of services under one endpoint/key or create a new resource for each service.

Cognitive Services multi-service account

Create Cognitive Services if you want to use multiple services in your application.

This one endpoint/key pair (multi-service) support:

  • Decision
    • Content Moderator
  • Language
    • Language Understanding (LUIS)
    • Language
    • Translator
  • Speech
    • Speech-to-text
    • Text-to-Speech
    • Speech translation
  • Vision
    • Computer Vision
    • Custom Vision
    • Face
  • Form Recognizer
  • Metrics Advisory

The Cognitive Services resource is available in price S0 (so, it’s not free) and not with all regions.

  1. Create Cognitive Services

Select (Unique) Name, Subscription, Location (Region), Pricing tier (for now, it’s only S0), select Resource Group ) and accept terms (https://www.microsoft.com/en-gb/licensing/product-licensing/products?rtc=1 ).

And after a few seconds, you can enjoy your API.

Now, you can grab your key and endpoint and test them in your application.

Specific Cognitive Services

The creation of specific Cognitive Services is almost the same.

You are going to choose the service you would like to use:

  • Azure OpenAI account
  • Speech service
  • Language Service
  • Translator
  • Language understanding (classic)
  • QnA Maker (classic)
  • Computer Vision
  • Custom Vision
  • Face API
  • Anomaly Detector
  • Content Moderator
  • Personalizer
  • Health Insight

The difference is that you can choose the F0 tier – a free, limited tier.

For example, Personalizer has a limit of 50K transactions per month):

Test your API

If you don’t have your application written yet, but you would like to test and see what kind of result will back, you can use APIs services: https://westcentralus.dev.cognitive.microsoft.com/docs/services/

For example, Computer Vision V3, Describe Image https://westeurope.dev.cognitive.microsoft.com/docs/services/computer-vision-v3-ga/

Tested image:

And results:

Application examples

GitHub Co-pilot

Microsoft Designer

Seeing AI App

Bing Visual Search

Anomaly Detector Interactive Demo

Resources