Play — or narrate — the 2024 Arup deepfake case: a Hong Kong finance worker joined a video call with what appeared to be the company's CFO and other executives. Every face and voice on the call was a deepfake. He wired US$25.6 million. Ask students to write, individually and silently, one sentence: 'If I had been on that call, what would I have looked for to catch it?' Take 3–4 hands. Do NOT correct them yet — you'll return to their answers at the end of class after they've learned why most of those signals no longer work. This primes the AP skill of Analyzing Risk (Skill Category 1): they must estimate likelihood/impact when the usual cues are unreliable.

1. 6m

   What makes an attack 'AI-augmented' — and why it matters

   Content

   An AI-augmented attack is any attack where generative AI does part of the work an attacker used to do by hand — writing the lure, cloning the voice, scraping the target, choosing the pretext. The change is not that attacks are new in kind; phishing, pretexting, and impersonation are decades old. The change is on three axes. First, cost: an LLM drafts a fluent, in-context email in seconds for fractions of a cent, versus hours of human writing. Second, skill floor: off-the-shelf tools mean an attacker no longer needs to be a strong writer, a native English speaker, or a skilled programmer — the model supplies those. Third, scale: with automated reconnaissance feeding an LLM, one attacker can send 10,000 individually personalized lures overnight (personalization at scale), where previously they could send one generic blast to 10,000 or one hand-crafted spear-phish to one target — not both.

   Delivery

   Emphasize that the threat model shift is quantitative, not just qualitative — attackers can now do what only nation-states could do five years ago. Ask: 'If cost drops 100× and skill floor drops, what happens to the number of attackers willing to try?' (Answer: it goes up sharply — this is the Analyze Risk framing.) Head off the misconception that 'AI attacks require elite hackers.' They don't; ChatGPT-style tools and commercial voice cloners lower the bar so much that unskilled criminals can run campaigns that used to require APT-level talent.

2. 6m

   AI-generated phishing — the red flags that stopped working

   Content

   Traditional phishing training taught students to look for: bad grammar, awkward phrasing, generic greetings ('Dear Customer'), obvious spelling errors, and mismatched tone. An LLM eliminates every one of these. Given a scraped LinkedIn profile and a two-sentence prompt, the model produces a message with correct grammar, the target's real manager's name, references to a real recent project, in the manager's normal writing style. Which cues survive? Structural ones, not linguistic ones: (1) the sender domain (look-alike domains like rn instead of m, or a Gmail address for an internal request), (2) unexpected urgency combined with a request to break policy — wire funds, buy gift cards, share MFA codes, (3) links whose visible text differs from the actual URL, (4) any request to move to a different, less-monitored channel ('text me on WhatsApp'), and (5) the request itself being one the real sender would never make through this channel. Worked example: an email arrives, perfectly written, from 'ceo@yourc0mpany.com' (zero, not O), praising the recipient's recent Q3 deck and asking for an urgent vendor payment. The prose gives nothing away. The domain and the pretext do.

   Delivery

   Walk students through the worked example on the slide side by side — traditional clumsy phish vs. AI-polished phish. Ask: 'Which of the classic five red flags still fire?' (Grammar/spelling/greeting: none. Domain, unexpected request, urgency + policy break: all still fire.) This directly targets AP Skill Category 3, Detect Attacks — the analysis skill of triaging cues. Correct the misconception aloud: 'If it reads well, it's legit' is now dangerous.

3. 5m

   Deepfakes and voice cloning — verify by channel, not by senses

   Content

   Voice cloning today needs roughly 3–30 seconds of clean audio — trivially obtained from a podcast, a keynote, a voicemail greeting, or a TikTok. Real-time video deepfakes on a Zoom call are harder but demonstrably in the wild (Arup, 2024). Subtle artifacts still exist — mismatched lip sync on plosives, unnatural blink rate, flat prosody on emotional words, lighting that doesn't shift when the head turns — but relying on the human eye to catch them is a losing bet, and it gets worse every quarter. The reliable defense is architectural, not perceptual: out-of-band verification. If the CFO calls asking for a wire, you hang up and call the CFO back on their known desk extension. If a video-call request breaks policy, you confirm via a pre-agreed code word or via a separate authenticated channel. The channel the request arrived on cannot verify itself.

   Delivery

   State this rule bluntly: 'Your ears and eyes are no longer sufficient authentication.' Ask: 'Why doesn't calling the number the email gave you count as verification?' (Because the attacker controls that number — verification must use a channel established before the attack.) Pre-empt the misconception that 'video calls are safe because I can see them' — the Arup case is your counter-evidence. This is AP Skill Category 2, Mitigate Risk: choosing a control that does not depend on human perception.

4. 4m

   The pipeline: recon → personalize → deliver, all automated

   Content

   The end-to-end AI attack chain has four stages. (1) Automated reconnaissance: bots scrape LinkedIn, company sites, GitHub, breach dumps, and social media to build a profile per target — role, manager, recent projects, writing samples, vendors used. (2) Generation: an LLM ingests the profile plus a pretext template and outputs a tailored lure — email, SMS, or a script for a voice-cloned call. (3) Delivery: automation sends at optimal times, from look-alike domains, with tracking. (4) Interaction: if the target replies, an LLM-driven agent can carry on the conversation in real time, escalating pressure, answering questions, and adapting. The point students must grasp: this is a pipeline, and each stage has been individually automated. That is why defenses that assume 'attackers can't afford to spear-phish everyone' are obsolete.

   Delivery

   Walk the four boxes and connect back to the earlier economics point — every stage that used to require a person now runs on cheap tokens. Ask: 'At which stage does the defender have the best chance to break the chain?' (Recon: reduce public data exposure. Delivery: DMARC/SPF/DKIM, look-alike-domain monitoring. Interaction: out-of-band verification. There is no single choke point — layered defense is required, Skill Category 2.)

1. 12m

   Activity 1 — Side-by-side phishing triage (AP FRQ-style)

   Targets AP Skill Category 1 (Analyze Risk) and Skill Category 3 (Detect Attacks). Students work in pairs for 8 minutes, then 4 minutes of whole-class debrief. Hand out the two messages below (or push to student devices). They annotate each and answer three FRQ-style prompts. Walk around and check that pairs are distinguishing linguistic cues (dead) from structural cues (alive). Debrief by cold-calling two pairs to defend their answer to prompt C. Student handout — Phishing Triage Message A (traditional): - From: security-alert@paypa1-support.co - Subject: Your acount will be suspend!! - Body: Dear Costumer, we detect unusaul activity on you're account. Click here immediatly to verify or you're acount will be terminate in 24 hour. Thank for you cooperation. — PayPal Security Team Message B (AI-augmented spear-phish): - From: dana.mercer@yourc0mpany.com (recipient: J. Alvarez, Accounts Payable) - Subject: Re: Nakamura Foods — updated wire instructions before Friday close - Body: Hi Jamie, quick one before I jump into the 2pm board prep — Legal signed off on the Nakamura Foods contract this morning and their treasurer just sent updated bank details (attached). Can you get the first milestone payment of $184,200 out today so we don't slip past the Friday close I promised Priya? I know the timing is tight, sorry. If you have any issue with the routing number, ping me on Signal, my desk phone is a mess today. Thanks — Dana Prompts (answer in complete sentences): A. List every red flag you can find in Message A. For each, state whether an LLM-written version of the same attack would still show that red flag. Be specific about why it would or would not survive. B. Message B is grammatically perfect and contextually plausible. Identify at least three structural indicators (not linguistic ones) that should still make Jamie suspicious. Justify each. C. Determine the single most reliable control Jamie's company can deploy that would defeat Message B even if Jamie fails to notice any of the indicators in part B. Explain in 2–3 sentences why this control works when human detection fails.

   Materials

   • Printed or on-screen handout (content below)
   • Pen

   Example outputs

   • Prompt A example: 'paypa1-support.co uses a 1 instead of an l — this is a look-alike domain and would SURVIVE an LLM rewrite because the domain is chosen by the attacker, not written by the model. Spelling errors like acount, unusaul, immediatly would NOT survive — an LLM produces clean spelling. The generic Dear Costumer would NOT survive — an LLM given a scraped name writes Hi Jamie.'
   • Prompt B example: three structural indicators — (1) look-alike domain yourc0mpany.com uses a zero instead of the letter O; (2) request combines unexpected urgency (today, before Friday close) with a policy break (new bank details from an email attachment, wire before verification); (3) attempt to move communication to a less-monitored channel (Signal) with a plausible excuse (desk phone is a mess).
   • Prompt C example: 'Mandatory out-of-band verification for any change to vendor payment details — Jamie must call Dana on her known internal extension (not any number in the email) AND independently confirm the new bank details with Nakamura Foods via a phone number from the original signed contract. This works because it does not depend on Jamie noticing anything — the process itself blocks the payment until a second, pre-established channel confirms.'
2. 13m

   Activity 2 — Tabletop: The Cloned CFO CallLab

   Targets AP Skill Category 2 (Mitigate Risk) primarily, with Skill Category 1 (Analyze Risk) in the debrief. Groups of 3–4. 9 minutes of group work, 4 minutes of share-out. Each group must produce a written verification procedure that would defeat the scenario. Emphasize: no proposed control may rely on the target 'listening carefully' or 'watching for lip-sync.' If a group proposes one, push back with 'and if the deepfake is good enough that they miss it — what then?' Groups that finish early may look up the C2PA content-authenticity standard (https://c2pa.org) and add a note on whether provenance signing would have helped here. Student handout — Scenario: The 4:47 PM Call You are the security lead at Meridian Logistics, a mid-sized freight company (450 employees). At 4:47 PM on a Friday, Ana Reyes in Treasury receives a WhatsApp voice call. The caller ID shows an unknown number, but the voice is unmistakably CEO Marcus Okafor. Marcus says he is in a taxi heading to LAX, that a confidential acquisition of a competitor is closing tonight, and that Ana must wire $1.9 million to an escrow account in the next 40 minutes or the deal collapses. He says Legal will send paperwork Monday, that only he and the CFO know about the deal, and that Ana is not to discuss it with anyone including the CFO. He sounds stressed but coherent. The voice, cadence, and even a small verbal tic (he says 'right, right' twice) are exactly Marcus. Marcus does a lot of public podcasts. Your task: Part 1 — Analyze the risk (3 min). - List the AI capabilities the attacker likely used. Cite evidence from the scenario for each. - Rate likelihood of success if Ana has no verification policy (low / medium / high) and justify. Part 2 — Design the verification procedure (5 min). Write a numbered procedure Ana should follow. It must: - Include at least one out-of-band verification step, with the specific channel named - Include at least one control that does NOT depend on Ana perceiving anything about the voice - Handle the 'don't tell anyone' constraint (which itself is a red flag) - Fit within the 40-minute time pressure — a procedure that takes 3 hours has failed Part 3 — Layered defenses (1 min). Name one control at each of these layers that Meridian should have had in place BEFORE this call happened: - Policy layer - Technology layer - Training / human layer Rule: no step in your procedure may be 'Ana listens carefully for deepfake artifacts.'

   Materials

   • Scenario handout (below)
   • Whiteboard or shared doc per group
   • Computer with browser (for optional evidence gathering)

   Example outputs

   • Part 1 example: 'Voice cloning (evidence: podcasts provide ample training audio; the verbal tic right, right suggests the model captured his idiolect). Automated reconnaissance (evidence: attacker knew Ana's role in Treasury, the CEO's name, and used a plausible pretext of an acquisition). Social engineering pressure stack (urgency, authority, secrecy). Likelihood if no policy: HIGH — every linguistic and perceptual cue is compromised, and the secrecy clause blocks Ana's natural instinct to check with the CFO.'
   • Part 2 example procedure: 1. Do not initiate the wire. State: I need to follow our verification protocol before any transfer over $50,000. 2. End the call. 3. Call Marcus back on his known internal extension stored in the company directory (not any number from the WhatsApp call). 4. If he does not answer, call the CFO on their known extension — the secrecy clause is itself a policy violation and does not override verification. 5. Require a signed dual-authorization form in the payment system before the wire can be released; the system enforces this regardless of what the caller said. 6. Log the incident with SOC even if it turns out to be legitimate. Part 3 example: Policy — dual-authorization + mandatory callback for wires over a threshold. Technology — payment system that enforces the callback field before release; caller-ID spoof filtering on corporate lines. Training — annual tabletop that specifically simulates a deepfake CEO fraud so employees have practiced saying no to authority under pressure.

1. An employee receives an email that is grammatically flawless, addressed to them by name, references their actual recent project, and asks them to approve a routine-looking invoice through a link. Which of the following BEST explains why the traditional phishing red flags they were trained on are unreliable here? A. The attacker paid a professional writer to draft the message. B. Generative AI can produce fluent, personalized messages at near-zero cost using data from automated reconnaissance, eliminating the linguistic cues but not the structural ones. C. Modern email clients auto-correct spelling errors in inbound mail. D. The employee's spam filter has been disabled. (Targets Skill Category 1, Analyze Risk.)

   multiple choiceB. Generative AI enables personalization at scale — an LLM fed a scraped profile drafts a fluent, contextual message for fractions of a cent. Linguistic red flags (grammar, generic greetings, awkward tone) disappear, but structural ones (sender domain, unexpected policy-breaking request, link mismatch, channel switch) remain. A is wrong because cost/scale rule out human writers per target. C and D are fabrications about client/filter behavior.

2. A CFO calls the accounting team by phone at 5:30 PM asking for an urgent wire transfer of $340,000 to a new vendor. The voice sounds exactly like the CFO's, including her usual phrasing. Describe the specific verification response the accountant should take, and explain in 2–3 sentences why this response works when the accountant cannot tell the voice is synthetic. (Targets Skill Category 2, Mitigate Risk.)

   short answerThe accountant should end the call and call the CFO back on her known internal extension (from the company directory, NOT any number provided during the suspicious call), and require dual authorization in the payment system before release. This works because it does not depend on the accountant perceiving anything about the voice — verification happens on a separate, pre-established channel that the attacker does not control, so even a perfect voice clone cannot pass the check. The channel that carried the request cannot verify the request.

3. A small nonprofit's board argues, 'We're too small a target for AI-based attacks — those are for big companies.' Analyze this claim: identify TWO specific reasons AI shifts the economics of attack such that small organizations are now realistic targets, and recommend ONE control the nonprofit should adopt that does not depend on employees detecting AI-generated content. (Targets Skill Categories 1 and 2.)

   short answerReason 1: cost per personalized lure has dropped to near zero (an LLM plus scraped LinkedIn data produces tailored spear-phishing at fractions of a cent), so attackers can profitably target small orgs where they previously could not. Reason 2: the skill floor has collapsed — off-the-shelf tools like commercial voice cloners and chatbots mean unskilled criminals, not just APTs, can run convincing campaigns, expanding the pool of attackers. Recommended control: mandatory out-of-band callback verification for any financial transfer or credential change over a low threshold, using phone numbers from a pre-established internal directory — this control succeeds regardless of whether the employee notices the attack.

4. A security team is evaluating three proposed controls against AI-augmented social engineering. Which control is MOST effective specifically because it does not rely on human perception of AI artifacts? A. Quarterly training that shows employees examples of deepfake videos so they can learn to spot lip-sync errors. B. An enforced policy requiring callback verification on a directory-listed number for any request involving funds transfer, credential reset, or vendor bank-detail change. C. An email banner that warns 'This message is from an external sender.' D. Encouraging employees to reply to suspicious emails asking the sender to confirm they are real. (Targets Skill Category 2, Mitigate Risk, and Skill Category 3, Detect Attacks.)

   multiple choiceB. Callback verification on a pre-established directory number defeats the attack even if the employee is completely fooled by the AI content, because it authenticates through a channel the attacker does not control. A depends on human perception, which is exactly what AI defeats. C helps only marginally and does nothing against internal-looking look-alike domains. D is actively dangerous — replying on the same channel lets an LLM-driven attacker continue the conversation in real time and reinforce the deception.

• ☐Printed handout for Activity 1 (Messages A and B + three prompts) — one per student or pair
• ☐Printed handout for Activity 2 (Scenario: The 4:47 PM Call + three parts) — one per group
• ☐Whiteboards or shared docs — one per group of 3–4
• ☐Student devices with browser (optional, for C2PA lookup in Activity 2)
• ☐Projector / slide deck
• ☐Pens