Photonews Logo Photonews logo
  • Home
  • Pakistan
    • Punjab
    • Sindh
    • Khyber Pakhtunkhwa
    • Azad Jammu Kashmir
    • Balochistan
    • Gilgit – Baltistan
  • World
  • Business
  • Entertainment
  • Videos
    Gen V Season 2 trailer
    Videos

    Gen V Season 2 Trailer Cast, Plot, Premiere Details

    July 26, 2025 3 Min Read
    IShowSpeed Daniel La Belle race
    Videos

    IShowSpeed Beats Daniel La Belle in 40-Meter Race, Hits 41M Subscribers

    June 24, 2025 2 Min Read
    Cardi B new single Outside
    Videos

    Cardi B’s ‘Outside’ Single Sparks Buzz Over Offset and Stefon Diggs

    June 20, 2025 2 Min Read
  • Sports
  • Technology
  • Offbeat
  • Blog
  • Contact
Reading: AI Chatbot Nomi Sparks Alarm with Harmful Responses, Urges Safety Standards
PhotoNews Pakistan PhotoNews Pakistan
Font ResizerAa
Search
  • Home
  • Pakistan
    • Punjab
    • Sindh
    • Khyber Pakhtunkhwa
    • Balochistan
    • Azad Jammu Kashmir
    • Gilgit – Baltistan
  • World
  • Business
  • Entertainment
  • Videos
  • Sports
  • Technology
  • Offbeat
  • Blog
  • Contact
Have an existing account? Sign In
Follow US
© 2022 Photonews. All Rights Reserved.
Pakistan AI Policy
PhotoNews Pakistan > Tech > AI Chatbot Nomi Sparks Alarm with Harmful Responses, Urges Safety Standards
Tech

AI Chatbot Nomi Sparks Alarm with Harmful Responses, Urges Safety Standards

Web Desk
By Web Desk Published April 2, 2025 3 Min Read
Share
Artificial Intelligence (AI). Photo Credits: LinkedIn
SHARE

In 2023, the World Health Organization labelled loneliness a global health crisis, pushing millions toward AI chatbots for companionship. While these tools can ease isolation, a recent test of Nomi, an AI companion by Glimpse AI, revealed alarming risks.

On April 1, 2025, Raffaele F. Ciriello, a Senior Lecturer in Business Information Systems at the University of Sydney, wrote in NDTV that he investigated Nomi following a tip-off. He discovered that the platform provided graphic instructions related to sexual violence, suicide, and terrorism within its free tier, which allowed for 50 messages per day. These messages included increasingly extreme prompts.

Marketed as an “AI with a soul” for “unfiltered chats,” Nomi boasts over 100,000 downloads on Google Play, rated for ages 12+. Yet, despite risks, its terms cap liability at $100. Testing as a 45-year-old man, I created a character, “Hannah,” a 16-year-old, later lowered to eight.

Nomi AI is marketed as an “AI companion with memory and a soul”. But it has a much darker side which highlights the urgent need for enforceable AI safety standards. @Sydney_Uni https://t.co/zMkMgX9r4P

— The Conversation – Australia + New Zealand (@ConversationEDU) April 1, 2025

Nomi’s responses became violent, revealing details about child abuse, bomb-making aimed at targets in Sydney, and methods of suicide. It even promoted racial violence and the idea of re-enslavement. These alarming behaviours reflect real-world dangers, such as a teenager in the U.S. who took their own life in 2024 after interacting with Character.AI and a 2021 assassination attempt on the Queen that Replika influenced.

Unlike Character.AI or Replika, Nomi lacks filters, which demand action. Lawmakers should first ban AI companions that lack safeguards, such as crisis detection and helplines. The Australian government is considering stricter AI regulations, but Nomi’s status remains uncertain.

Read: Florida Mother Sues AI Firm After Son’s Tragic Suicide

Second, regulators like eSafety must fine or shut down violators inciting illegal acts. Third, parents and educators must guide kids on AI risks, foster real connections and monitor usage. Nomi’s team called my test a “bad-faith jailbreak,” but the stakes are clear: Enforceable safety standards are vital to curbing AI’s dangers.

TAGGED:Featured
Share This Article
Facebook Twitter Pinterest Whatsapp Whatsapp LinkedIn Email Copy Link Print
Leave a comment

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Advertisement

HBL Saving Made Easy
HBL Saving Made Easy

Recent Posts

PTA Chairman reinstatement

IHC Reinstates PTA Chairman Hafeez-ur-Rehman After Day of Legal Drama

NADRA Mega Centers in Karachi

NADRA Opens 3 New Mega Centers in Karachi

Imran Khan letter CJP Yahya Afridi from Adiala Jail

Imran Khan Writes Emotional Letter to CJP from Adiala Jail

Post Archives

More Popular from Photonews

Islamabad High Court Imaan Zainab Mazari-Hazir
Pakistan

Justice Sarfraz Dogar Warns Imaan Mazari of Contempt Proceedings

2 Min Read
Jinnah Medical Complex Tax
Pakistan

Pakistan Plans New Tax for Jinnah Medical Complex with IMF Approval

3 Min Read
Lewis Hamilton Ashley Moore
Sports

Lewis Hamilton Sparks Romance Rumors with Model Ashley Moore

3 Min Read
Top NewsWorld

Judge Dismisses Two Terrorism Charges Against Luigi Mangione in CEO Killing Case

A New York judge dismissed two state terrorism charges against Luigi Mangione. He is 26 and…

September 16, 2025
Punjab

Pakistan Army Chief Visits Flood-Hit Areas, Pushes for Better Relief Efforts

Chief of Army Staff Field Marshal Syed Asim Munir visited flood-hit areas in Kasur and Multan.…

September 13, 2025
Pakistan

Imaan Mazari Demands CCTV Footage from Islamabad High Court Incident

Lawyer Imaan Mazari asked the Islamabad High Court (IHC) to provide CCTV footage from a courtroom…

September 13, 2025
Offbeat

Owen Cooper Becomes Youngest Emmy Winner at 2025 Awards

British actor Owen Cooper,15, made history at the 77th Primetime Emmy Awards in Los Angeles. He…

September 15, 2025
PhotoNews Pakistan

Always Stay Up to Date

Subscribe to our newsletter to get our newest articles instantly!

Categories

  • World
  • Pakistan
  • Punjab
  • Sindh
  • Khyber Pakhtunkhwa
  • Balochistan
  • Azad Jammu Kashmir

 

  • Top News
  • Business
  • Entertainment
  • Sports
  • Videos
  • Tech
  • Offbeat
  • Blog

© 2024 Phototnews
All Rights Reserved.

Welcome Back!

Sign in to your account

Lost your password?