Skip to main content

Generative Media: Image, Audio & Video AI

Beginner

Most of AILmanac is about AI that writes and reasons — text models like Claude. But a whole other branch of AI makes media: pictures, video, voices and music from a prompt. This page is a provider-neutral map of that landscape: the archetypes, the notable tools (each verified to exist), how they differ from text LLMs, the skills that carry over, and the rights-and-ethics rules you can't skip.

What you'll learn
  • Know the main archetypes of generative media — image, video, voice/music, speech-to-text — and a real tool in each
  • Understand how media models differ from a text chat model like Claude
  • Learn the durable skills: prompting for media, iteration, and reference inputs
  • Respect the non-negotiables: usage rights, provenance/watermarking, and deepfake/consent ethics

How media models differ from a text LLM​

If you've only used a chat model, four differences matter:

  • Different output, different "correct." A text model returns one mostly-deterministic best answer. A media model returns an artifact (an image, a clip, a voice take) where there is no single right answer — only ones you like more or less. You generate several and pick.
  • You usually can't edit the output by editing the prompt alone. Tweaking one word can change the whole image. Real control comes from reference inputs (a starting image, a style image, a reference voice) plus tool-specific controls, not just words.
  • Seeds and variation are part of the workflow. Many tools let you re-roll, vary, or lock a "seed" for reproducibility. Iteration is the norm, not a failure.
  • The cost unit is different. Text bills by tokens; media bills by image, second of video, or character/minute of audio — and video especially can get expensive fast.
Pro tip
  • Mental model: a text model is a writer who gives you the answer; a media model is a slot machine with a steering wheel — you pull, then steer with references and re-rolls until it's right.
  • Generate a batch and curate. Your taste — knowing which of 4 outputs is best — is half the skill.

The map: archetypes and notable tools​

Think in archetypes — the lineups shuffle constantly, the categories are stable. Every tool named below was verified to exist with a real URL (in Sources); specific versions are intentionally left vague because they churn.

Image generation (text-to-image)​

Turn a text prompt — and optionally reference images — into a still picture.

  • Midjourney — closed, hosted; known for a strong default aesthetic and style/reference controls.
  • OpenAI image generation (the DALL·E / GPT Image line) — closed, hosted; tightly integrated into ChatGPT and the API.
  • Google's Gemini image generation ("Nano Banana") and Imagen — closed, hosted; wired into Gemini and Google Cloud.
  • Adobe Firefly — closed, hosted; built for creative workflows and trained with commercial use in mind (see rights, below).
  • Stable Diffusion (Stability AI) and FLUX (Black Forest Labs) — open-weight model families you can download and run yourself, the backbone of the local/open image ecosystem.

Video generation (text/image-to-video)​

Generate short clips — increasingly with synchronized audio — from a prompt or a starting image.

  • OpenAI Sora — a frontier video model (note: the standalone consumer app was discontinued; the model continued via API — a textbook example of why versions/availability go stale).
  • Google Veo — frontier video, with native audio in recent versions.
  • Runway — a creative suite built around its Gen-series video models.
  • Kling (by Kuaishou) and Pika — fast-moving video tools popular for social/short-form content.

Voice, speech & music (audio)​

  • ElevenLabs — closed, hosted; lifelike text-to-speech and voice tools (voice cloning has real consent implications — see ethics).
  • Suno and Udio — generate full songs (vocals + instrumentation) from a text prompt.
  • OpenAI Whisper — the other direction: speech-to-text (transcription). It's open-weight (MIT-licensed), so you can run it yourself or call a hosted API. Speech-to-text is the quiet workhorse behind captions, voice notes, and meeting transcripts.
Pro tip
  • The single biggest fork (same as for text models): CLOSED hosted tools (often the best quality, least setup) vs OPEN-WEIGHT models like Stable Diffusion, FLUX and Whisper that you can self-host for privacy, control, and lower cost at scale.
  • Don't pick by leaderboard. For your actual look/sound, generate a few real examples in 2-3 tools and judge with your own eyes and ears.

How to get a good result from a media model​

The workflow is more like a photo shoot than a Google search. These steps are tool-agnostic:

Guided walkthrough1 of 6
  1. Lead with the main subject, then add setting, action, and mood. 'A red fox' is weak; 'a red fox curled asleep in fresh snow, soft morning light' gives the model something to work with.

Example image-generation prompt (medium → subject → style → details)

A cinematic wide-angle photograph of a lighthouse on a rocky cliff
at golden hour, waves crashing below, dramatic storm clouds parting.
Style: moody, high dynamic range, shot on a 24mm lens, shallow depth of field.
Mood: hopeful, epic. Aspect ratio 16:9.

Tip: if the result is close, change ONE element next pass
(e.g. "blue hour" instead of "golden hour") and re-roll —
or add a reference image to lock the look.

Rights, provenance & ethics — the non-negotiables​

Media AI raises issues text chat usually doesn't. Treat these as part of the job, not an afterthought.

Usage & commercial rights. Every tool has its own license, and they differ a lot. Some restrict commercial use, some grant it only on paid tiers, and outputs can resemble copyrighted or trademarked work. Adobe Firefly, for instance, leans on training data chosen with commercial safety in mind — but you should still read the actual license for the tool and tier you're on rather than assume.

Provenance & watermarking. A growing standard, C2PA / Content Credentials, attaches tamper-evident "nutrition label" metadata recording how an asset was made and edited. Many major image and video tools now embed it. Keep that provenance intact, and prefer tools that support it — it's becoming the norm for trustworthy media.

Deepfakes, voices & consent. Realistic face- and voice-generation can impersonate real people. Don't clone a voice or likeness without explicit consent, never use generated media to deceive, and follow the law in your jurisdiction. Voice-cloning consent and disclosure of AI-generated content aren't just etiquette — increasingly they're legal requirements.

Watch out
  • Generated media can carry usage-rights limits and deepfake/consent risks — check each tool's license and disclose AI-generated content.

What transfers from your text-AI skills​

Good news: a lot of what you already know carries over.

  • Clear, specific prompting — being concrete about what you want beats vague wishes, in every modality.
  • Iteration over one-shot — the same "draft, critique, refine" loop you'd use with Claude applies, just with re-rolls and reference images instead of follow-up messages. See Prompting basics.
  • Choosing the right tool for the task — the framework from Choosing a model (constraints first, then a tiny real-world test) works just as well for media.
  • Understanding what the model actually is — knowing these are pattern-learners, not magic, keeps your expectations and your ethics straight. See Foundations.

Check yourself​

Check yourself

0/4
  1. What's the most reliable way to control the look of a media model's output — beyond words?
  2. Which of these is an OPEN-WEIGHT tool you can download and run yourself?
  3. What is C2PA / Content Credentials?
  4. Before using AI-generated media commercially, the safest move is to:

Flashcards​

Press Enter or Space to flip the card. Use the left and right arrow keys to move between cards.Term shown.
1 / 6
Key takeaways
  • Generative media is a separate branch of AI from text chat: it makes images, video, voice and music, and you steer it with references and re-rolls, not just words.
  • Think in archetypes — image, video, voice/music, speech-to-text — and remember the closed-vs-open-weight fork (e.g. Midjourney vs Stable Diffusion/FLUX; hosted TTS vs open Whisper).
  • The durable skills transfer from text AI: specific prompting, iteration, and choosing the right tool for the task.
  • Rights and ethics are part of the work: check the license for your tool and tier, keep provenance (C2PA/Content Credentials), get consent for voices/likenesses, and disclose AI-generated media.
  • Specifics rot fast — verify versions, prices and capabilities against each tool's own page before relying on them.

Sources & further reading​