GenAI Concepts
About this resource
This page offers basic GenAI terms and examples, developed in collaboration with the ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S) and the Office of the Victorian Information Commissioner (OVIC).
To cite: Fan Yang, Jake Goldenfein, and Kathy Nickels, ‘GenAI concepts’, ADM+S and OVIC (Web Page, 2024), https://www.admscentre.org.au/genai-concepts/
Technical
You will find basic GenAI terms, descriptions, and examples in this section.
Generative AI or GenAI
Prompt and prompt engineering
Machine learning
Large language models (LLMs)
Knowledge cut-off date
Chatbot
General purpose AI
Foundation Model
Frontier AI
Transformer architecture
Transfer learning
Open-source and closed-source LLMs
Token
Reinforcement learning from human feedback (RLHF)
Diffusion models
Inference
Operational
In this section, you will find the development, distribution, and deployment of GenAI systems in real-world applications.
Distribution
Supply chains
AI libraries
Machine learning environments
Deployment and Use
Fine-tuning
Deployer or ML/ops
User
Development
Datasets
Data licensing
Developer or dev
Data labelling and annotation
Distribution
Supply chains
AI libraries
Machine learning environments
Deployment and Use
Fine-tuning
Deployer or ML/ops
User
Regulatory
While GenAI offers numerous opportunities, this section will take you through risks and concerns to be addressed through regulations, ethical guidelines, and practices.
Chatbot
Diffusion models
Foundation Model
Frontier AI
Generative AI or GenAI
General purpose AI
Inference
Knowledge cut-off date
Large language models (LLMs)
Machine learning
Open-source LLMs and Closed-source LLMs
Prompt and prompt engineering
Reinforcement learning from human feedback (RLHF)
Token
Transfer learning
Transformer architecture
To cite:
Fan Yang, Jake Goldenfein, and Kathy Nickels, ‘GenAI concepts’, ADM+S and OVIC (Web Page, 2024), https://www.admscentre.org.au/genai-concepts/
Technical
You will find basic GenAI terms, descriptions, and examples in this section
Generative AI or GenAI
GenAI is short for Generative artificial intelligence or Generative AI. These are software systems that create content including text, images, music, audio, and videos based on a user’s ‘prompts’.
Types of GenAI models in the market
Prompt and prompt engineering
A prompt is an instruction, query, or command that a user enters into a GenAI interface to request a response from the system.
Note that because GenAI systems produce texts through statistical predictions of the most appropriate words in a sentence, the responses that GenAI systems produce may not always be the same. This is why GenAI outputs are sometimes described as non-deterministic.
Prompt engineering is the practice of writing inputs for GenAI tools so that the tool can produce optimal outputs.
Example:
Basic structure of a prompt:
Acting as a [role] perform [task] in [format] in the [style]
Acting as a [role] | Create a [task] | Show as [format] | In the [style] |
Each role would provide a context and a background of the information. | The task should be clear and specific. | It shows how you would like to structure the information generated. | This is optional. It shows what tone you’d like to use for the information. |
Marketer Advertiser Copywriter Accountant Lawyer Financial analyst English professor Journalist Project manager Manager Engineer Recruiter … | Headline Presentation Webinar Blog post Book outline Email sequence Social media campaign Product description Cover letter Summary TikTok, YouTube and Instagram Real - Video script Sales page / ad copy … | A table A list With bullet points Summary HTML Code Spreadsheet CSV file Plain text file Rich text Markdown Word Cloud … | Formal Poetic Enthusiastic Shakespearian Accessible Basic English Scientific Objective Neutral Pop culture … |
As a [life influencer ( the role)], create a [blog post (the task)] about the benefits of daily exercsie in a [PDF (the format)], with an [accessible and enthusiastic tone (the style)]
Tip:
If the task is too complicated, break it down into steps to make it easier for the chosen Gen-AI to read and interpret.
Machine learning
Machine learning is a set of techniques for creating algorithms so that computational systems can learn from data.
A machine learning algorithm is a set of rules or processes that helps an AI system to perform specific tasks, such as finding patterns in data or making predictions based on the inputs. In this way, the model’s behaviour reflects the data or the learning experience.
Higher-quality data helps algorithms improve their accuracy in various tasks, whether it be recognising faces in photos, predicting the weather, or recommending products to buy.
Types of machine learning
- Supervised machine learning:
An algorithm is provided with labelled data, for example, a collection of pictures of animals with labels telling which animal is in each picture. The algorithm then learns from these examples and tries to predict the correct labels for new, unseen data. - Unsupervised machine learning:
An algorithm is provided with unlabelled data and then it tries to find patterns or structures in the data on its own. - Hybrid machine learning:
This type of machine learning combines elements of both supervised and unsupervised learning approaches as well as potentially other techniques to leverage the strengths of different methods.
Applications:
- Healthcare: ProMed (the Program for Monitoring Emerging Diseases) offers an online real-time data analysis and reporting system showing outbreaks of infectious diseases worldwide.
- Finance: Fraud detection, automating trading activities, and providing financial advisory services to investors.
- Marketing: Product recommendations and news feed on social media services.
- Transportation: Self-driving vehicles, real-time tracking, and last-mile delivery optimisation.
Large language models (LLMs)
Large language models are data transformation systems. They are trained with large numbers of *parameters. When a user inputs a prompt, the model will generate text content in response.
LLMs are trained on extremely large datasets sourcing data from websites, blogs, forums, and news articles etc. They contain millions or billions of parameters. While GPT-1 contains 117 millions parameters, GPT-2 contains 1.5 billion parameters, and GPT-3 contains 175 billion parameters. Increased parameters often lead to more complex models that can handle more complicated tasks and generate nuanced text.
Users interact with the GPT model through interfaces like ChatGPT. This feedback loop allows the model to continuously learn and improve over time based on user feedback.
Examples:
Some notable LLMs are OpenAI’s GPT series of models (e.g., GPT-1, GPT-2, GPT-3, GPT-3.5, and GPT-4) used in ChatGPT and Microsoft Copilot.
-
- Google’s PaLM and Gemini
- xAI’s Grok
- Meta’s LLaMA family of open source models
- Antropic’s Claude models
Related term: ‘Machine learning‘
Knowledge cut-off date
A knowledge cut-off date of a GenAI model is the date when the training data for a specific large language model was last updated. It defines the limitations of the model’s understanding and knowledge.
Example:
ChatGPT-3.5’s current knowledge cut-off date is January 2022. The knowledge cut-off date for ChatGPT-4 is April 2023.
Chatbot
A popular application of GenAI and LLMs are chatbots. A chatbot is a computer program that interacts with humans through natural language conversations. Some chatbots use LLMs to generate content according to user inputs.
Chatbot functionality is often embedded within a service, i.e., banking, tax, logistics, ecommerce etc., to work as customer services. These chatbots might be fine-tuned on an organisation’s private datasets, so that it can answer specific queries.
Example:
Apple’s Siri, Amazon Alexa, Google assistant, ChatGPT, Virtual assistants to provide customer services in applications.
General purpose AI (GPAI)
General purpose AI or GPAI is a new paradigm where very large AI models become competent at a wide range of tasks, often without substantial modification or fine-tuning. General purpose AI systems are often large language models.
Some general purpose AI systems can be effective at processing various forms of information inputs.They can process audio, video, textual and physical data, and even complex medical or scientific data with enough training. Part of the commercial appeal of Foundation Models are their capacity for general purpose applications.
Example:
*GPT3, GPT4, BLOOM, BERT, stable diffusion, and Dall-E. They are designed for general purposes and are used for downstream natural language processing tasks.
Foundation model
Foundation models, sometimes called ‘general purpose AI systems’, provide a basis for future modification and fine-tuning for specific tasks.
Foundation models are trained on vast amounts of data at scale including images, texts, audios, videos, or other data types, with less emphasis on data quality so that they can be adapted to a wide range of downstream tasks.
Examples:
Several companies including Open AI, Google, Meta, and Stable Diffusion own foundation models. Many open -source foundation models can be downloaded from Hugging Face, GitHub, TensorFlow Hub, and PyTorch Hub for further deployment. Popular foundation models include GPT, BERT, and DALL-E 2.
Related terms: ‘Machine learning‘, ‘Transformer architecture’, ‘Open source models’, ‘General purpose AI’, and ‘Large language models’.
Frontier model
Frontier models are larger models with more parameters than existing foundation models. They are potentially much more capable than existing models and raise additional safety risks.
Challenges:
There are different definitions of Frontier models that use different criteria. These definitions are not necessarily exclusive to one another but rather adopt approaches to assessment, potentially leading to various policy discussions and interventions.
- Based on risks:
The UK government defines frontier AI as consisting of highly capable general-purpose AI models that can perform a wide variety of tasks and match or exceed the capabilities present in today’s most advanced models (see AI Safety Summit: introduction).
The paper, entitled Frontier AI regulation: Managing emerging risk to public safety by staff from Google, Open AI, and Centre for Long Term Resilience defines frontier AI models as highly capable foundation models that could possess dangerous capabilities sufficient to pose severe risks to public safety.
- Based on the computational power used:
The training of the frontier AI model requires computer power exceeding 10^26 integers or floating-point operations (FLOPs) (see Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence).
- Based on both risks and computational power required:
The EU AI Act categorise a general-purpose AI model as having systemic risk if it has high impact capabilities and the cumulative amount of computation used for its training measured in FLOPS (floating point operations) is greater than 10^25 (Article 51, Section 1 Classification rules, CHAPTER V GENERAL-PURPOSE AI MODELS).
Transformer architecture
The transformer architecture makes large-language models (LLMs) possible, including ChatGPT (Chat Generative Pre-trained Transformer). It was initially published by Google in 2017 in the paper ‘Attention is All You Need’.
Software processes information by tokens and handle sequences step by step. Transformer models process long sequences simultaneously and capture context of the input. They allow developers to train larger networks with more training data at a given scale.
Examples of transformer architecture:
- Google’s BERT
- OpenAI’s GPT
- Meta’s BART
More information:
The central idea of the transformer architecture is attention and self-attention.
Attention: When encoding a sentence, the transformer architecture pays attention to each of the words, figuring out which ones are needed to understand the whole sentence and where they appear, giving these higher attention scores.
Self-attention: A mechanism to contextualise words by paying attention to other words that make up its context in a body of text.
Visit Tranformers: The Google scientists who pioneered an AI revolution for a visualisation of this concept.
Transfer learning
Transfer learning is a model’s ability to apply information about one situation to another. In this way, the model builds on its internal knowledge.
In GenAI, it is important for products to be able to adapt and work well in different situations. It’s like teaching a model skills in one area and then letting the model use those skills in other areas too. In this way, models can learn from one thing and apply it to lots of different tasks, making them versatile and useful.
Example:
A current trained model knows how to distinguish dogs and cats from other animals based on a lot of data labelling and data annotation. Now you want your model to identify a new animal, for example rabbits. You don’t need to start fresh as cats, dogs, and rabbits do share similar features like fur, legs, ears, etc but with some differences. You need to fine-tune the model to let it recognise rabbits. The model should be able to apply what it has learned about dogs and cats, the fine-tuned datasets, and apply the knowledge to recognise rabbits. Transfer learning makes the training process faster.
Relevant terms: ‘Machine learning‘, ‘General purpose AI‘, and ‘Large language models‘.
Open-source and closed-source LLMs
Open-source LLMs have their source code and underlying architecture publicly accessible, allowing developers, deployers, researchers, and enterprises to use, modify, and distribute them freely.
Closed-source LLMs have their source code and underlying architecture proprietary. They are accessible only under specific terms defined by their developers.
Open-source LLMs and closed-source LLMs can be accessed and deployed via provided *APIs (application programming interfaces).
The table below presents different features of open-sourced LLMs and closed-source LLMs.
Open-source LLMs | Closed-source/proprietary LLMs |
Descriptions | |
Open-source LLMs often means that a project team has made their model architectures, source code, and *weight parameters publicly accessible, on source code repository hosting platforms such as GitHub, GitLab, and BitBucket | Close- source or proprietary LLMs do not disclose the source code, how the model is structured, the datasets used to train the system, or the training process. These models are often developed as commercial products and may require licenses or subscriptions for their use. Most frequently, we can access the model through the provided APIs and use the model as provided by the owner. |
Features | |
Other programmers or developers can read the code, audit for bugs or security flaws, and potentially contribute improvements back to the project. | These models typically cannot be examined, audited, or modified. |
Examples | |
In general, we can think about these open-source projects like public toolkits of different models for developers and users to build their own tools. Access to open source LLMs can unlock customisation opportunities. OpenAI’s GPT models: The models of GPT-2 and GPT-3 are able to generate human-like text based on input prompts. The code and pre-trained models are available on GitHub, allowing developers/deployers to experiment with and build upon them.
| Data labelling is more complicated than annotation. It used to identify key features present in the data while minimising human involvement. The real-world use cases include NLP, audio and video processing, computer vision etc. by Google: Gemini Pro is open to developers and enterprise customers of Google. It can be accessed via the Gemini API. |
Advantages | |
Free to use - a lot of open-source LLMs are free of charge.
| Legal protection - companies and startups that use closed-source LLMs usually have legal agreements and terms of service that provide legal protection for their businesses.
Scalability - with the company’s resources, closed source LLMs can be scaled more efficiently. Regular update - with resources gained from the proprietary model, LLMs can be constantly maintained and updated.
|
Concerns | |
There are different ways of doing open source and the term is contested. However, for GenAI, there are a group of open-source models and ecosystems that will have their own development, procurement, and deployment pathways.
| Vendor dependency - companies using closed-source LLMs can be dependent on the AI or Machine Learning development company. Challenges may come when the development company discontinues support.
|
Token
Token is the smallest unit of data used by GenAI systems.
For text-based models like GPT, a token is often a word, a part of a word, punctuation marks, spaces, or other elements of the text. For image-generating AI models like DALL-E, a token is a pixel of the image. For audio-based AI models like MusicLM, a token might represent a short segment of sound.
Tokens are fundamental to allow AI models to comprehend, memorise, and generate meaningful responses. They play a vital role in memory capacity, determining how much information the AI model can recall before forgetting.
Example:
Different versions of ChatGPT come with different memory capacities. For example, the ChatGPT Plus plan offers up to 8,000 tokens. This means that ChatGPT Plus can only remember information within the last 8,000 tokens during ongoing interactions with the user.
More information:
Counting tokens is especially important when LLMs bill input and output tokens differently. For example, with Anthropic’s Claude 3 Opus model, the cost for one million input tokens is $15 USD and the cost for one million output tokens is $75 USD.
Both input (the prompt or conversation history) and output (the model’s response) tokens contribute to token usage. That is, if the input message uses 10 tokens and the model’s response generates an additional 15 tokens, then the user, whether an individual or an organisation, will be billed for a total of 25 tokens.
Reinforcement
learning from human feedback
(RLHF)
Reinforcement learning from human feedback (RLHF) resembles the human learning process. When we learn a skill, our teacher/instructor says things like “well done” when we do something right or “let’s try it again and differently” to improve our skill.
Application:
Reinforcement learning from human feedback works a bit like this. Human testers, trainers, or programmers observe how AI software conducts tasks and provides feedback through ranks and scores.
The model tries different actions based on the feedback and learns which actions are more preferable and which ones are not based on the scores. Over time, with enough feedback and practice, the model gets better at conducting tasks, making decisions, and achieving its goals even without human involvement.
Example:
The system might provide two possible outputs to a prompt, and the human indicates which one they prefer.
Diffusion models
Diffusion models are used for AI image generation. They work by destroying training data (i.e., by adding visual noise) and then learning to recover the data by reversing the noise.
Application:
Imagine you have a picture of a dog. A diffusion model would take this picture and add some random changes or destroy a part of it, like making some parts of the picture blurry or changing some colours (i.e., adding noise). Then, the model’s job is to figure out what the original picture looked like before noise was added (i.e., denoising). Diffusion models learn from lots of examples to figure out how to undo the damage and reveal the original image.
Examples
DALL-E 2, Midjourney, and Stable Diffusion use diffusion models.
Inference
AI Inference is the process of applying trained machine learning models to new, unseen data to derive meaningful predictions or decisions. When users give a GenAI system a prompt, the computational system used to produce the output is called inference. The energy required for inference is much lower than training a model, but is still significant and is a large part of the cost of using a GenAI system.
Applications:
Almost any real-world application of AI relies on AI inference. Some of the most commonly used examples include:
Large language models (LLMs) – a model trained on sample text can parse and interpret texts it has never seen before.
Predictive analytics – once a model has been trained on past data and reaches the inference stage, it can make predictions based on incoming data.
Email security – A machine learning model can be trained to recognise spam emails or business email compromise attacks, then make inferences about incoming email messages, allowing email security filters to block malicious ones.
Operational
In this section, you will find concepts related to development, distribution, and deployment of GenAI systems in real-world applications.
Development
Datasets
Large Language Models (LLMs) are trained from different types of datasets. The material in datasets may be protected by intellectual property or information privacy laws. Organisations like Common Crawl scrape the public internet periodically and create gigantic datasets that can be used for model training.
Application:
Datasets used for fine-tuning models are typically made available through platform marketplaces like Hugging Face, AWS, and Azure. These datasets are often made available on commercial terms, and typically have more information about how the dataset was created, and whether relevant legal rules were complied with.
Most AI companies, like OpenAI, trained their data on the open web without seeking permission under the Fair Use terms in the US. Using datasets for AI model training can be a controversial subject.
Data licensing
Data licensing is like giving permission for others to use the data. There are different types of data licenses that dictate what others can and cannot do with your data, such as whether they can share it, modify it, or use it for commercial purposes.
Examples:
Organisations with large desirable datasets like Reddit or New York Times are seeking financial compensation for licenses to train AI models on their data.
On the other end, AI companies have their arguments against paying for copyrighted content.
- Adobe claims that they license all the copyright material used to train their commercial models.
- Google claims that AI training is like reading a book. If training could be accomplished without the creation of copies, there would be no copyright questions.
- Microsoft claims that changing copyright law could hurt small AI developers.
Developer or Dev
AI developers belong to a broader category of programmers or engineers. They write code and algorithms to enable machines to perform tasks that normally require human intelligence. They build models in-house by training on a mix of public and private data for various applications from chatbots and virtual assistants to self-driving cars.
In emerging AI regulations, developers are responsible for creating AI software products and complying with particular regulations.
Example:
The EU AI Act imposes different obligations on AI ‘developers’, ‘deployers’, and ‘users’. Many of the regulatory obligations around safety and auditing must be satisfied by developers before their product can go to market.
With organisations and individuals assuming distinct while overlapping roles within the AI supply chain, third-party organisations have emerged, offering services like Future of Life’s The EU AI Act Compliance Checker to allow individuals and organisations to identify their roles and obligations under certain legislation.
Data labelling and annotation
Data labelling and annotation are terms used to describe the process of tagging or labelling the data available in the forms of text, images, audio, and video for AI training. The differences are described below.
Data annotation | Data labelling |
Meaning | |
Data annotation is the practice of adding relevant *metadata to a set of collected data so that the machine learning algorithm can easily recognise the dataset. | Data labelling is to attach some meaning to different types of data to train a machine learning model. It identifies a single entity from a set of data. |
Purpose | |
Data annotations can be used for visual based perception models. Data annotation can help recognise relevant data through *computer vision. | Data labels are used to identify dataset features for NLP algorithms. Data labelling is used for training advanced algorithms to recognise patterns within the datasets in the future. |
Applications | |
Data annotation is a fundamental element in creating training data for computer vision. Annotated data is required to train machine learning algorithms to see the world humans see. | Data labelling is more complicated than annotation. It used to identify key features present in the data while minimising human involvement. The real-world use cases include NLP, audio and video processing, computer vision etc. |
Distribution
Supply chains
An AI supply chain refers to the process of creating, sourcing, and integrating the various components needed to develop and deploy AI systems or products.
Concerns:
Understanding AI supply chains requires us to ask questions like:
Where the data used to train the AI comes from;
whether the AI model is open for anyone to use or owned by an entity, the software used to develop the AI;
and how organisations tweak the AI using different data sources and annotations.
Often, developers use pre-made software parts, which makes it challenging to know who’s responsible for following the rules and regulations.
AI libraries
AI libraries host a collection of pre-written codes and algorithms for common AI tasks, such as data preprocessing, model training, and evaluation.
Examples:
- PyTorch is one of the most popular machine learning libraries alongside TensorFlow. It offers free and open-source software released under the modified BSD license.
Development: PyTorch was originally developed by MetaAI and is now part of the Linux Foundation umbrella.
Notable applications: Tesla Autopilot, Uber’s Pyro, and Hugging Face’s Transformer are built on top of PyTorch.
- TensorFlow is a free and open-source software library for machine learning and artificial intelligence. It can be particularly used for training and interference of deep neural networks.
Development: TensorFlow was developed by the Google Brain team.
Notable applications: GE Healthcare, Twitter’s ranking system, and the image-captioning software DeepDream
- Hugging Face ecosystem, including the Transformers and the Hugging Face Hub, contains libraries for tasks such as dataset processing, model evaluation, and machine learning demos. The Hugging Face ecosystem is most notable for its transformers library built for natural language processing applications. Hugging Face Hub also allows users to share machine learning models and datasets.
Development: Hugging Face, Inc is a French-American company that originally developed a chatbot app targeted at teenagers.
Notable applications: with the partnership with Amazon Web Services (AWS), Hugging Face’s products are available to AWS customers and can be used as building blocks for their custom applications.
Machine learning environments
Machine learning environments are where machine learning models are built, trained, and deployed. Provided by the big tech corporations like Amazon and Microsoft, these environments contain fully managed infrastructure and tools to enable reproducible, auditable, and portable machine learning workflows across different *compute targets.
Examples:
Examples of different types of machine learning environments:
- Microsoft’s Azure machine learning environments.
- Amazon’s SageMaker Studio and SageMaker Studio Classic.
Deployment and Use
Fine-tuning
Fine-tuning happens during model development and deployment. The process involves modifying an already trained AI model with a smaller targeted fine-tuning dataset. The aim of fine-tuning is to maintain the original capabilities of a pretrained model while adapting it to suit more specialised use cases.
Example:
Imagine that you have a model that’s been trained to recognise birds in pictures, but you want it to be better at recognising Swift Parrots specifically. You decide to deploy the model that you found from an AI library, fine-tune the model by feeding it lots of pictures of Swift Parrots, adjusting its *parameters, and even scoring the performance (i.e., reinforcement learning) of the computer until it gets better at identifying them.
More information:
Large training datasets are often available from development platforms like Github, Azure AWS, Papers with Code, and Hugging Face.
The Data Provenance Initiative identified that across Github, Hugging Face, and Papers with Code, more than 70% of pretrained datasets had no data licenses; those that did, roughly half were incorrect and were more permissive than the dataset creators had intended (read more in Public AI Training Datasets are Rife with Licensing Errors and The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI).
Deployer or ML/ops
While AI developers create the AI system, AI deployers take the AI systems or models that have been developed and make them available for use in real-word applications. In some cases, the distinction between a developer and a deployer is minimal. However, deployers may be the organisations that make AI tools available to others.
Applicatons:
Let’s think about an AI model being composed of four layers: infrastructure, model development, application development, and applications. Different levels are open to deployment but are associated with different levels of technical capacities.
The New AI Stack
From the table above, AI deployment can be as simple as a dashboard at the level of application or as complex as the augmentation of a foundation model through prompting engineering, or specific generative AI architecture at the level of infrastructure.
From the pie chart above, open-source AI tools are deployed for the purposes of coding, communications (i.e., WhatsApp bots, Slack bots), information aggregation (e.g. requesting the model for meeting summaries), image production, workflow automation, and writing.
Source from https://huyenchip.com/2024/03/14/ai-oss.html.
User
Users are the individuals or organisations that interact with AI technology, whether through text, voice, images, or other forms of input. Users can also be anyone whose rights and activities are influenced by AI systems’ outputs.
Examples:
We are users when we chat with our virtual assistant like Siri or Google Assistant.
Even though individuals do not actively engage with AI models, it is possible that individuals’ rights and activities can be affected by organisations’ incorporation of AI models. For example, when AI is used to screen job applications and select candidates for interviews. Outcomes are likely to be influenced by the AI decision-making system.
Regulatory
This section will take you through risks and concerns to be addressed through regulations, ethical guidelines, and practices.
Risks
AI regulation is emerging as a type of risk-based regulation. This approach to regulation imposes obligations depending on the risk level of the system. Most regulatory proposals have three risk categories – ‘low risk’, ‘medium risk’, and ‘high risk’. The measurement and mitigation of AI-generated risks requires human oversight, AI auditing, transparency, and explainability.
Challenges:
Measuring risks is not an easy task! Risk is a spectrum and depends on a number of factors that are difficult to quantify. There are standards being developed to help developers, deployers, and users specify the appropriate level of risk for the AI system.
Concerns:
The use of AI to play chess would generate low risks compared to using AI for staff recruitment which would generate medium risks, the latter of which can lead to gender and / or racial discrimination. The use of AI-enabled robots for medical surgery and in self-driving cars to make real-time decisions would generate high risks as such risks are systematic, perpetual, or hard to reverse.
Privacy and data protection
The wide use of AI systems presents challenges to privacy. The datasets used to train foundation models and other machine learning systems often include personal information without consent. There have been cases where chatbots reproduce personal information from training data in response to prompts. Privacy and data protection laws impose obligations on entities that collect and process ‘personal information’.
These obligations relate to the collection, use and disclosure of personal information; data quality and security; and the deletion, minimisation or deidentification of personal information. Protecting individuals’ data often requires that individuals consent to the use of their personal information, or limiting uses to only those that an individual would reasonably expect, and that relate to the original reason for collection. Privacy and data protection laws also create rights for individuals to have access to information about them held by other entities.
Legislation:
In Victoria, individuals’ privacy rights are protected by the Privacy and Data Protection Act 2014 (Vic) (PDP Act). The PDP Act contains 10 Information Privacy Principles (IPPs) which are the core of privacy law in Victoria and set out the minimum standard for how Victorian public sector organisations should manage personal information.
Under the PDP Act, personal information is information or an opinion about you where your identity is clear or where someone could reasonably work out that it is related to you. It can include your name, email address, postal address, phone number, signature, fingerprint, photographs or surveillance footage of you, comments written about you (whether true or not), or your financial details, and more (see more at Your Privacy Rights).
Challenges:
We’ve learned that AI models are trained on large datasets and our real-time inputs are sometimes included in the AI training process. Organisational data and personal and sensitive information may therefore be inadvertently disclosed to the companies in control of the AI software being used.
A combination of seemingly non-personal information can make individuals identifiable when analysed or correlated.
Further, AI’s predictive capabilities can exceed what an individual discloses, resulting in the collection of new personal information generated by the AI model’s output (read more at Artificial Intelligence – Understanding Privacy Obligations).
Sometimes, AI systems can be used to discriminate against someone based on their race, ethnicity or gender.
More information:
The Office of the Australian Information Commissioner reported a 19% increase in data breaches from July to December 2023, with notable impacts in sectors such as health services, finance, insurance, retail, and the Australian government, where our sensitive information is stored (read more in Notifiable Data Breaches Report July to December 2023).
If an organisation is using generative AI services that include accessing a model through an API (application programming interface), then the organisation should ensure that those data flows do not breach relevant privacy and data security legislation, such as the Victorian Information Privacy Principles (IPPs) or Australian Privacy Principles (APP). This will require assurance from the software provider around how their data is stored, processed, and secured.
Hallucination
Hallucination refers to AI models making up fictional facts to fit the prompt’s intent. When an LLM processes a prompt, it is searching for the statistically most appropriate words, not necessarily the most accurate answer. This answer might sound very convincing but has no basis in fact. This creates very significant risks for organisations that rely on chatbots to give advice about the products or services they provide, because that advice might not be accurate.
Real case:
In 2023, two lawyers in New York used ChatGPT in their legal research for a personal injury case. Their legal brief, submitted to the court, included six fake case citations generated by ChatGPT (i.e., hallucination). The lawyers conducted a fact-check with ChatGPT which approved the results generated. The two lawyers and the law firm received penalties as a result.
Concerns:
AI systems are not tasked with telling the truth. Sometimes people may mistakenly think that AI systems only get things wrong occasionally while otherwise telling the truth. That is not true. We need to understand that AI systems make mistakes so it is important to verify the accuracy of the output before relying on the model. This is especially important when people rely on AI systems to make decisions that affect themselves or others.
Tip:
One way to identify AI-generated images is by looking at the detail (i.e., hands, teeth, hairs, unnaturally smooth skin, accessories, watermarks in the background, and stereotypes). AI systems often struggle with those complex details and are constructed based on visual stereotypes, leading to odd or unrealistic results (i.e., hallucination).
Safety
In AI regulation, AI safety means that AI should be designed, developed, and used in a way that is safe, human-centric, trustworthy, and responsible (see The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023).
The main concern of AI safety is to make sure that AI systems are developed, deployed, and used in ways that align with the norms and values of the broader public or specific user groups. This is also to ensure that AI systems work along with diverse human interests.
Guidelines and legislation:
AI safety is central to numerous laws and AI ethics guidelines developed at national and supranational levels, such as The Bletchley Declaration, the EU AI Act, The Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence.
In Australia, the government is already undertaking work to strengthen existing laws, for example Online Safety Act 2021, to address known harms with AI.
Along with ethical guidelines including Australia’s Artificial Intelligence Ethics Framework and Australia’s AI Ethics Principles, the Australian Government’s interim response to safe and responsible AI consultation describes actions to be taken to help ensure that AI is safe and responsible.
Measurement:
Safety measures can include testing AI systems thoroughly before using them, designing them to be robust enough to handle unexpected situations without making mistakes, and having guardrails in place.
Transparency
AI transparency is a fundamental regulatory and ethical requirement. Transparency can have different meanings for different groups of stakeholders (e.g., developers, deployers, users, regulators, and so on).
Transparency in AI is important because it represents how well users, regulators, and other stakeholders understand how the system operates, and how it was built. This is important for ensuring trust in the outputs of AI systems.
Transparency is often related to both the openness and explainability of AI systems.
Varieties:
Different stakeholders will be interested in different forms of transparency.
As general public users, transparency can be a sense of what the AI system is doing and why.
For a regulator, transparency can enable auditing how a prediction and decision is made by AI systems in detail, or mean transparency in the data used to train a system.
Engineers can be more interested in the internal mechanism and parameters of AI systems.
For high-risk AI products or services, transparency of known risks and steps taken to mitigate those risks is critical for all stakeholders involved.
Related term: ‘Explainable AI‘
Copyright
Copyright law typically applies to original works of authorship, such as literary works, artistic works, and computer programs. Copyright material has been included in AI training datasets without permission or rights clearance.
Cases:
There are currently court cases and policy reforms addressing how to manage the inclusion of copyright material in datasets. There is also the risk that GenAI responds to prompts with outputs that violate copyright. This might be an exact reproduction of a text or image, or the unauthorised use of a copyrighted character.
Several coders have brought a lawsuit against GitHub, Microsoft, and OpenAI as GitHub Copilot was trained and developed on lines of open-source code that had been written leading to questions of attribution.
Several visual artists filed a class-action lawsuit against the companies of Stable Diffusion, Midjourney, and DreamUp all of which generate images based on text prompts from users. The case alleges that AI tools violate copyrights by scraping images from the internet to train the AI models.
Getty Images alleges that Stable Diffusion’s use of its images to train models infringes on copyrights and some of the images even contain the Getty watermark.
Many providers of GenAI systems will now indemnify users in case an AI output breaches copyright.
Challenges:
The application of copyright to datasets raises complex issues. The ability of rights-holders to opt out or demand licensing fees will depend on how the law develops, and will vary in different jurisdictions. For instance, some jurisdictions may create exceptions for model training, and others may determine that the training process does not replicate the ‘expressive’ part of a copyright work, and thus is not a breach.
Data governance and security
Data governance refers to the system of rules for management and control of data.
Data governance requirements may involve compliance with laws, standards or ethical guidelines to ensure data within an organisation is collected, stored, and used appropriately.
Within an organisation, data governance responsibilities may include ensuring data quality (i.e. that data is accurate) as well as data security (i.e. that data is appropriately encrypted and protected from unauthorised access. This may mean additional care when using organisational data to fine tune a GenAI system, or when using organisational or sensitive data as part of a prompt.
Data lifecycle management refers to the processes for managing data from creation to disposal, including data archiving and removal.
Legislation:
The Australian Government will develop a whole-of-government Data Governance Framework for public services to ensure data quality, data privacy, data authority, and data innovation. So far, there are sector-based data governance frameworks, for example, by Australian Commission on Safety and Quality in Health Care, Australian Institute of Health and Welfare, National Archives of Australia, and Human Research Ethics.
In Victoria, there is the Victorian Protective Data Security Framework and Standards and Privacy and Data Protection Act 2014.
AI standards
International AI standards serve as a global governance mechanism to help achieve AI policy goals. Standards organisations create standards through stakeholder input, and cover various technical and regulatory dimensions of AI systems. Compliance with certain standards may be required for AI products to be market-worthy, or procurers may also require compliance with specific standards. In the EU system, if developers comply with standards then they likely comply with their obligations under the AI Act.
Applications:
A wide range of organisations engage in the ongoing development of AI standards.
International standards bodies, including ISO/IEC, IEEE, and the ITU have been engaging in the development of AI standards.
-
ISO/IEC 42001:2023 – Information Technology – Artificial Intelligence – Management system standards is the world’s first AI management system standard that provides valuable guidance for this rapidly changing field. It addresses the challenges posed by AI technologies including ethical considerations, transparency, and continuous learning. For organisations, the standard sets out a structured way to manage risks and opportunities associated with AI, balancing innovation with governance.
-
ISO/IEC 23894:2023 – Information technology – Artificial intelligence – Guidance on risk management provides guidance on how organisations that develop, produce, deploy, or use AI products, systems, and services can manage risks related to AI.
-
ISO/IEC 23053:2022 – Framework for Artificial Intelligence (AI) Systems using Machine Learning (ML) establishes an AI and ML framework for describing a generic AI system using ML technology.
-
IEEE is an engineers’ professional organisation with a subsidiary Standards Associations (SA) whose standards address protocols for digital products including Ethernet and WiFi. Its AI standardisation processes are part of a larger IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems.
-
The ITU has historically played a role in standards for information and communications technologies particularly in telecommunications. Following the 2018 AI for Good Global Summit, the ITU has created a number of Focus Groups on AI in relation to health, environmental efficiency, autonomous and assisted driving, and data.
-
As part of European AI regulation, the European Commission has asked the European Committee for Standardisation (CEN) and the European Electrotechnical Committee for Standardization (CENELEC) to develop harmonised technical standards for companies that wish to comply with the EU AI Act.
Specialised bodies may also participate in creating international standards. These bodies can be treaty organisations such as the International Atomic Energy Agency and the International Civil Aviation Organisation.
More information:
There are more than 1,000 AI policies and principles developed by governments, international organisations, and private entities.
In Australia, there are Australia’s Artificial Intelligence Ethics Framework and Australia’s AI Ethics Principles. The Australian Government’s interim response to Safe and Responsible AI in Australia Consultation highlights that more needs to be done to ensure the safe and responsible development and deployment of AI.
Guardrails
AI guardrails are a set of predefined standards, limitations, and operational protocols to prevent AI systems from making decisions or taking actions that could lead to harmful or unintended consequences.
Guardrails are often sets of algorithms or rules that filter and edit both the inputs (i.e. the prompts) and the outputs of a GenAI system to ensure that the outputs comply with legal and safety requirements – for instance, ensuring that the outputs do not breach copyright, produce political misinformation, create biased or discriminatory information, or generate hate speech.
Jailbreaking:
Jailbreaking refers to techniques for getting around a system’s guardrails. This might mean tricking a chatbot into generating outputs that would otherwise be a violation of its policies. Most LLM systems are susceptible to jailbreaking, meaning all guardrails have limited efficacy.
Regulatory sandbox
Regulatory sandboxes are software testing environments where businesses can conduct limited testing of innovations and test their regulatory status.
This allows businesses to receive feedback from experts, including regulators, investors, innovators, and other stakeholders, regarding the potential and viability of the innovation.
Application:
Regulators and innovators use regulatory sandboxes to offer individualised legal guidance. Sometimes, regulatory sandboxes are used to make legal exceptions for certain innovations, enabling innovators and the public to experience the impact of new technologies as if they had already been established as safe and effective.
Legislation:
Regulatory sandboxes have been included in the EU AI Act Article 53 and Australian Securities & Investments Commission.
Human oversight
Human oversight is the requirement that human actors oversee the output of AI systems to ensure that the system creates accurate and accountable results. There are different forms and degrees of human oversight, depending on the context and purpose of the AI system in use. Human oversight is usually required in cases of high risk systems.
Application:
Humans can intervene in AI systems in different forms and at different degrees.
Humans in the loop: even though AI systems can do a lot of tasks on their own, humans can be part of the AI system to make decisions, provide approvals, or improve the system by correcting or preventing potential errors that an AI system can make.
Humans on the loop may not need a human to say yes or no every step of the way; there is still a person watching over the process to make sure everything runs smoothly.
Humans in command means that humans have the ultimate control over the AI systems. It is like being the commander of the AI system where the commander makes all the important decisions.
Sometimes human oversight requires a final human decision-maker, but often this is not the best way to implement oversight or may be impossible.
Legislation:
The EU’s AI Act Article 14 requires that high-risk AI systems have appropriate human oversight from design to operation. The level of human oversight should be proportional to the potential impact and severity of harm that the AI system can cause.
These rules have been referred to and explained in the documents below:
AI auditing
AI auditing involves humans, normally researchers, programmers, and regulators, looking closely at AI systems to evaluate the associated risks, ensure AI systems act fairly, safely, and comply with relevant laws, regulations, and ethical standards.
Application:
AI auditing is a rapidly growing field of industry practices. There are emerging courses, certifications, and associations aiming to professionalise the AI auditing industry.
There are different ways of AI auditing and they have different advantages and limits.
-
Technology-oriented audits focus on the properties and capabilities of AI systems.
-
Process-oriented audits focus on technology providers’ governance structures and quality management mechanisms.
Some auditing methods may be simple compliance checkboxes, others may be more comprehensive assessments of how AI systems might affect users and other stakeholders.
Explainable AI (XAI)
Explainable AI or XAI is a set of tools, techniques, and algorithms designed to produce high-quality interpretable, intuitive, human-understandable explanations of AI decisions. Many emerging AI regulations require some degree of explanations for the outputs of higher-risk systems.
Application:
The explainability of AI-driven decisions is often linked to the transparency, fairness, and trustworthiness of the decision made, and sometimes the interpretability of AI systems.
The goal is not to prioritise complete explainability over performance or vice versa. Organisations should disclose the limits of transparency available in the system. It’s about finding a balanced approach that considers the risks and benefits of each AI application, while also taking into account the human and environmental implications with the use of AI systems.
Post-market surveillance/monitoring
Post-market surveillance refers to ongoing monitoring of the performance and safety an AI product or service after it has been released to the market.
Application:
Post-market surveillance may involve collecting and analysing data on the use of the product or service in real-life conditions. If any issues are detected, the information gathered through post-market surveillance can help in taking corrective actions, such as updating product design, improving instructions for use, or recalling a product.
Examples:
In 2022, Meta (the parent company of Facebook) pulled the public demo of Galactica, a large language model designed for science research, three days after its launch due to its inability to distinguish facts from falsehood (i.e., hallucination).
In 2024, Google paused its Gemini AI model’s image generation of people due to inaccuracies in some historical depictions.
FLOPS
FLOPS means Floating Point Operations Per Second. In the context of computing and artificial intelligence, FLOPS is often used to measure the processing power or performance of hardware devices like CPUs (Central Processing Units) and GPUs (Graphic Processing Units).
Sometimes ‘frontier models’ are defined by FLOPS, and some regulatory proposals have stricter compliance obligations for systems that surpass FLOPS thresholds.
Higher FLOPS values indicate higher computational power, more complex AI models, faster data processing, better graphic performance, and higher energy consumption.
Application:
The US Executive Order on Artificial Intelligence specifies different regulatory requirements for systems that use different amounts of compute. It states that federal safety and security reporting requirements apply to any model trained using either:
a quantity of computing power greater than 10^26 FLOPS
or primarily biological sequence data and a quantity of computing power greater than 10^23 FLOPS. So far the threshold is beyond anything on the market.
The largest GPT-3 model required 3.14E+23 FLOPS for training with 174,600 parameters, and 300 billion training tokens.
Ecological
Training, deploying, and using AI systems contribute to the global CO2 emissions. Typically, more powerful AI models require more energy. The servers that power AI models also generate considerable heat, and are often water cooled. The amount of water used (i.e. evaporated) to train an AI model can be immense. OECD AI Policy Observatory estimated that around 50 queries of GPT-3 in Australia would evaporate around one litre of water.
Computation used to train notable AI systems
LLMs are among the biggest machine learning models, spanning up to hundreds of millions of *parameters, requiring millions of *graphics processing unit (GPU) hours to train, and emitting carbon in the process.
To cite: Fan Yang, Jake Goldenfein, and Kathy Nickels,’GenAI concepts’, ADM+Sand OVIC (Web Page, 2024), https://www.admscentre.org.au/genai-concepts/
Acknowledgement of Country
In the spirit of reconciliation we acknowledge the Traditional Custodians of Country throughout Australia and their connections to land, sea and community. We pay our respect to their Elders past and present and extend that respect to all Aboriginal and Torres Strait Islander peoples today.