Grok is an Artificial Intelligence (AI) image application that allows users to create or alter images from text prompts.
Grok is currently free to use, with some usage limits. The application can be used within the social media platform X. X’s terms and conditions state that the minimum age to use the service is 13.
AI applications such as Grok allow users to adapt existing images from text prompts. There is no requirement that the user is the author, owner or subject of the original image. Grok provides a “Spicy Mode” as part of paid subscriptions that allows the user to create partially nude and sexually suggestive content. This function in Grok and other apps is sometimes called ‘nudifying’ or ‘undressing’.
The images created by people using any AI application could be considered illegal in NZ if they cross a legal threshold and are found to be objectionable. Determining a classification is partly about what’s depicted, but also how it’s depicted, and its potential to cause serious harm.
Our information resource on banned or illegal content provides further detail about the content that is always classified as objectionable. Please note our resource includes references to distressing and harmful material.
Even if the content created is not objectionable, users that create false images of real people (deepfakes) might still be committing an offence under:
- the Harmful Digital Communications Act (2015)
- the Crimes Act (1961)
- the Harassment Act (1997)
- the Defamation Act (1992)
If you are concerned that a deepfake image has been created of you, contact Netsafe. Netsafe helps with online harm, bullying, harassment, image-based abuse, and privacy breaches. They provide free, confidential support and can help get harmful content removed.
It’s important for parents and whānau to know that Grok isn’t the only AI application that allows users to create content from text prompts. There are several applications that can do the same or similar thing and they may have limited safety measures in place to prevent the creation of potentially objectionable content under NZ law.
We know from conversations with young people that curiosity plays a role in coming across harmful content online. We also know that many people are unaware that the creation of objectionable content carries significant penalties and even image creation for ‘fun’ or curiosity could result in someone being charged with a crime.
Read our information resource on AI-generated content: When is it illegal in New Zealand?
Seen something harmful and need to talk to someone? We have a Helplines and Support Services Resource.
Seen something harmful and need to report it to the authorities? We have a Reporting resource for harmful or illegal online content in New Zealand.
Frequently asked questions (FAQs)
Why doesn’t the Classification Office ban Grok if it can be used to make objectionable imagery?
The Films, Videos, and Publications Classification Act 1993 (the Act) sets out the mandate for the Classification Office to classify publications as either objectionable (banned), restricted (e.g. to a certain age or group), or unrestricted.
The Act does not provide the Classification Office, or any other body, with the power to ban websites, software or online tools of any kind. The Act only applies to the classification of “publications”, i.e. specific pieces of content that have already been created. The Office can, and regularly does, examine and classify AI-generated content.
How do I keep my child safe if they are using an AI-application?
We recommend using parental controls on any devices and applications that your tamariki may use. These can include age restrictions, screen time limits, and passcodes. Our resource on parental controls provides broad advice on how to get started with different operating systems and software.
I’ve come across harmful images generated using AI, who do I report this to?
If you come across content you think may be illegal:
- Do not share it with others
- Do not save or forward it
- Report it as soon as possible
Useful links:
How to report harmful or illegal online content in New Zealand
About the Classification Office’s role and how the law works
The Films, Videos, and Publications Classification Act 1993 balances freedom of expression with protecting people from harm – especially children and young people.
Every piece of content the Classification Office sees is different and to decide whether something is illegal it must consider each one on its own merits and carefully apply the Act and the principles of fairness and natural justice. There are also serious penalties for offences under the Act. This is why decisions about whether content is illegal are made by trained experts at the Classification Office, who, guided by the Act, follow a very transparent and fair process before making a decision.
You can read more about the classification process here.
Was this helpful?
If you'd like to know more about this topic, get in touch. We're happy to help.
