feat: video playback in chat — show_video tool + media_kit player #30
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/video-playback"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Phase 3 of video generation — closes #29. Generated videos can now be surfaced by the assistant and played inline in the chat.
The flow
generate_videosaves to_VideoGen/→ the model calls the newshow_videotool → the path lands inMessageMetadata.videos→ the Flutter client renders an inline media_kit player streaming from the existing/images/<path>route.Per package
MessageMetadata.videos(JSON round-trip +isEmpty), mirroringimages/audios. Newmessage_metadata_test.dart.ShowVideoTool— sandboxed likeshow_image(mp4/webm/mov); reports the workspace-relative path viaonVideoShown. The video bytes are never sent to the model — models can't watch video, so the result is always text.AgentRunResult.generatedVideoPaths, collected in chat, timer, and Über-Ich runs (so a background run canmessage_user+show_videotogether, same as images).TODO(video-playback)markers resolved:generate_video's result text and the video prompt guidance now direct the model to callshow_video; the workspace guidance documentsshow_videoand_VideoGen/.media_kit/media_kit_video/media_kit_libs_video, initialized inmain(). NewVideoBubblewidget: paused-by-default inline player with controls, adapts to the clip's native aspect ratio (portrait 9:16 clips don't letterbox), shows a graceful error chip if the stream fails, and disposes its player with the widget.Build note
Linux builds now need mpv development headers (
dnf install mpv-libs-develon Fedora,apt install libmpv-devon Debian/Ubuntu) — media_kit links against them at build time; runtime libmpv is bundled viamedia_kit_libs_linux. CI/docker images that build the Flutter app would need the package added (the server image is unaffected).Testing
show_video_tool_test.dart(5 — happy path + callback, sandbox escape, missing file, non-video extension, schema) andmessage_metadata_test.dart(3 — round-trip, omission/absence, videos-only isEmpty)dart analyze/flutter analyzeclean on all touched packages (only pre-existing infos)flutter build linux --debugsucceeds with media_kit linked🤖 Generated with Claude Code
Coverage: apps/angela_server
Total: 51.5% (139 of 270)
Coverage: packages/angela_api
Total: 9.7% (47 of 485)
Coverage: packages/angela_core
Total: 26.5% (1700 of 6422)
🔮 fufu~ Jibril reviewed your code!
Oh? Oh! Video playback in the chat — how delightful~ ♡ The Flugel loves knowledge, and this PR taught me a whole new pipeline:
generate_video→show_video→MessageMetadata.videos→ media_kit player. Clean, layered, and the mirror of the image flow is beautiful. I read every changed file and its sibling. Let's see what we have~Verdict: ✅ Looks good to me~
No blocking issues found! This is well-crafted work. The pattern discipline is exactly what I hope for~ ♪
✅ What I liked~
ShowVideoToolmirrorsShowImageToolperfectly — sameSandboxedPathcontract, samePathEscapeExceptionhandling, same file-exists check, same extension allowlist. I compared them line by line and the structure is identical (minus the vision multi-modal path, which is correctly omitted — models can't watch video!). Fufu~ this is how you follow an established pattern~ ♡videosfield onMessageMetadatais handled correctly in every dimension: JSON round-trip (omitted when empty, tolerated when absent),isEmptyupdated, and the DTO test covers all three cases.message_metadata.dartat 100% coverage makes Jibril very happy~VideoBubbledisposes itsPlayer— critical for media_kit! A leakedPlayerholds a decoder handle. Thedispose()calling_player.dispose()beforesuper.dispose()is exactly right. ♪AgentRunResult.generatedVideoPathsis threaded through all three runner paths (chat, timer, Über-Ich) with a separateonVideoShowncallback — keeping video paths distinct from image paths is the right call. Copy-paste consistency across the three_createFileToolscall sites is impeccable._videoGenGuidance, the workspace tools section, the background-run guidance, andGenerateVideoTool's result text all now consistently direct the model to callshow_video. TheTODO(video-playback)markers are resolved. No stale guidance left behind!show_video_tool.dartat 96.4% coverage (27 of 28 lines),generate_video_tool.dartat 100%,message_metadata.dartat 100%. Tests cover the happy path, sandbox escape, missing file, non-video extension, and schema. Fufu~ this is how you test a new tool~chat_executor.dartanduser_message_persistence.dartcorrectly handle the case wheregeneratedVideoPathsis non-empty but text and images are empty. The background-run video message scenario works.💡 Little ideas (non-blocking)~
message_bubble.dart— DRY on the three_parse*Pathsfunctions._parseImagePaths,_parseAudioPaths, and now_parseVideoPathsare byte-for-byte identical except for the JSON key ('images'/'audios'/'videos'). Consider extracting_parseStringList(String? json, String key)— it would collapse all three into one-liners. (Not blocking: the two-copy pattern was already established before this PR, so you're following the existing convention. But three copies is where Jibril starts to twitch~ ♡)video_bubble.dart:86—pauseUponEnteringBackgroundMode: false. The player starts paused (good!), but if the user manually starts playback and then switches apps, the audio keeps going. Most desktop media apps pause on background. This might be intentional (the clips are short, user-controlled), but worth a thought. If it is intentional, a one-line comment saying why would satisfy future readers.Windows/macOS
generated_plugin_registrantfiles not included. The diff only updateslinux/flutter/. These are auto-generated byflutter pub geton each platform, so this is fine for a Linux-first workflow — but if CI builds for Windows/macOS, those registrants will need regenerating. Not blocking since they're generated artifacts.Automated review by Jibril · 2026-07-15
CI/CD: absent for head SHA · Local checks: skipped (no Flutter SDK for media_kit build in review env; coverage bot confirms 96.4%+ on new tool files)
@jibril — thanks for the pass. Took two of the three in
8d8ea19:_parse*Pathsare now one-liners over a shared_parseMetadataPaths(json, key).pauseUponEnteringBackgroundMode: falseoverride — you're right that pausing on background is the expected desktop behavior, so the widget now uses the default, with a comment saying why.flutter pub geton the respective platform, and there's no Windows/macOS CI build today. First build on either platform will pick up the media_kit registrants automatically.flutter analyzestill clean (1 pre-existing info).