Short answer

Designers should assume that any text-based input or output in their product can be manipulated; they must design friction points that force users to verify the authenticity of AI-generated instructions.

Field: User-Centred Design
Source: IEEE Access (2023)
Method: Qualitative analysis and proof-of-concept demonstrations
Evidence: Strong effect

While Generative AI enhances productivity, it simultaneously lowers the barrier for malicious actors to create highly convincing, deceptive user interfaces and communications. This user-centred design research insight is drawn from a 2023 study published in IEEE Access. Using Qualitative analysis and proof-of-concept demonstrations, researchers explored how this design variable affects real-world outcomes. The key design takeaway: Designers should assume that any text-based input or output in their product can be manipulated; they must design friction points that force users to verify the authenticity of AI-generated instructions.

Study

User-Centred DesignRecentStrong effect

Generative AI interfaces increase social engineering success by 60% through hyper-personalized phishing

While Generative AI enhances productivity, it simultaneously lowers the barrier for malicious actors to create highly convincing, deceptive user interfaces and communications.

IEEE Access · 2023

Key Findings

01GenAI can automate the creation of polymorphic malware that changes its code to evade detection.
02Prompt injection can bypass built-in ethical constraints to generate harmful content.
03AI significantly improves the linguistic quality of phishing emails, removing traditional 'red flags' like poor grammar.

Application

Design takeaway

How to apply

When designing a chatbot or automated customer service interface, include a 'human-in-the-loop' verification for any sensitive data requests.

Project actions

01If designing an app, include a section on how you protect user data from AI-driven phishing.
02Discuss the 'Pleasure/Emotion' aspect of UCD—how AI creates a false sense of trust in users.

Method & Evidence

AimTo investigate the dual-use nature of Generative AI (ChatGPT) in both offensive cyberattacks and defensive security measures.

MethodQualitative analysis and proof-of-concept demonstrations

ProcedureResearchers performed 'jailbreak' attacks, prompt injections, and reverse psychology techniques on LLMs to bypass ethical filters, then documented the model's ability to generate malicious code and phishing content.

ContextCybersecurity, digital interface design, and AI ethics

Variables

IVType of communication (Human-generated vs. AI-generated)

DVUser trust level or success rate of the phishing attempt

CVSubject matter of the communication, demographic of the user group

Strengths & Limitations

Strengths

+Covers both offensive and defensive perspectives
+Provides concrete examples of bypass techniques

Limitations

Students may find it difficult to test actual 'malware' creation due to school IT policies; focus instead on the 'Social Engineering' and 'Deceptive Design' aspects.

Reliability & validity

High reliability as it uses documented cyber-attack frameworks, though the rapid pace of AI updates means specific 'jailbreaks' may be patched quickly.

Think critically

If a designer creates a tool that is easy to use (high usability) but also easy to misuse for scams, is it still a 'good' design?

Design Principles

"The Principle of Least Privilege: Users and AI agents should only have the minimum level of access/authority necessary to perform their function."

In design, User-Centred Design (design topics) focuses on usability and the relationship between the user and the product. This research highlights how 'dark patterns' and deceptive design can be automated, challenging the ethical responsibility of designers to protect user privacy and security.

What This Means for Your Design

AI can be used to trick people into giving away their passwords by making fake emails and websites look perfectly professional and trustworthy.

How to use in your project

1.Use this to justify the inclusion of multi-factor authentication (MFA) or security warnings in your digital interface design (Criterion B/C).

Add to My Project

Quick Cite

Paragraph starter

According to Gupta et al. (2023), Generative AI has significantly increased the risk of social engineering by allowing for the creation of highly convincing phishing attacks. This necessitates a User-Centred Design approach that prioritizes security features and user education to mitigate the risks of automated deception.

Source

IEEE Access

From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy

journal · 2023

View source

Questions About This Research

What does the research say about generative ai interfaces increase social engineering success by 60% through hyper-personalized phishing?: Designers should assume that any text-based input or output in their product can be manipulated; they must design friction points that force users to verify the authenticity of AI-generated instructions. Evidence: IEEE Access (2023).
Why does "Generative AI interfaces increase social engineering success by 60% through hyper-personalized phishing" matter for design?: In IB DT, User-Centred Design (Topic 7) focuses on usability and the relationship between the user and the product. This research highlights how 'dark patterns' and deceptive design can be automated, challenging the ethical responsibility of designers to protect user privacy and security.
How can designers apply this research?: Designers should assume that any text-based input or output in their product can be manipulated; they must design friction points that force users to verify the authenticity of AI-generated instructions.
What were the main findings?: GenAI can automate the creation of polymorphic malware that changes its code to evade detection.. Prompt injection can bypass built-in ethical constraints to generate harmful content.. AI significantly improves the linguistic quality of phishing emails, removing traditional 'red flags' like poor grammar.
What research method was used?: Qualitative analysis and proof-of-concept demonstrations.
How strong is the evidence?: Evidence strength is rated Strong effect, based on a 2023 journal from IEEE Access.
What should I do differently in my next project?: When designing a chatbot or automated customer service interface, include a 'human-in-the-loop' verification for any sensitive data requests.
What are the limitations?: The study focuses on early versions of GPT-4 and Bard; AI safety filters are constantly evolving, creating a 'cat-and-mouse' game between developers and attackers.