akhaliq/Fibo-2.0-8step-smd

Fibo 2.0 — 8-step t2i LoRA (SMD distillation, WIP)

A work-in-progress 8-step, guidance-free text-to-image LoRA for briaai/Fibo-2.0-dev (30 steps, CFG 5.0), distilled with SMD — on-policy velocity regression (arXiv 2610.18472). No fake-score network, no adversarial training: the student runs its own 8-step Euler trajectory and regresses its velocity onto the frozen CFG-5 teacher's, one trajectory segment per update.

Status: v0.3 (update 5500) — ahead of edit-turbo on average, with visible caveats. On the 6-prompt benchmark this checkpoint scores LPIPS 0.467 vs dev-30-step, ahead of edit-turbo's 0.485. But the metric rewards tone and texture while geometry failures are local: the library staircase still dissolves into the bookcase at the top, and complex signs get a mirrored duplicate above the main text. For structural/architectural prompts and text-heavy scenes, edit-turbo remains the safer pick; for product shots, portraits, and stylized scenes, this student is the better t2i option. Judge by the grid, not the number.

Weights (primary)	update_5500/lora_student_ema.safetensors — rank-128 LoRA, fp32 (EMA), peft format, targets every attention to_q/k/v/out.0 + FFN w1/w2/w3 of the Fibo-2.0 transformer
Sampling	8-step Euler, mu-shifted sigmas (mu = 1.15 at 1024²), CFG 1.0 (no negative prompt). 4 steps does NOT hold (see sweep) — use 8
Teacher	Fibo-2.0-dev at CFG 5.0 with the model-card negative caption
Training data	1,200 structured VGL JSON captions (combinatorial templates; v0.2 widened lighting/palette vocabulary)
Updates	5,500 prompt-updates: 1,500 on one A100 (5.3 h) + 1,500 on 2×H200 DDP (76 min) + ~3,500 on 4×H200 DDP with batch-2 + flash-attn-3 (2.5 h); checkpoints every 500

v0.3 samples (update 5500)

History: u3000 grid at eval/grid_3way_u3000.jpg, u1500 grid at eval/grid_3way.jpg.

Checkpoint sweep (LPIPS vs dev 30-step, mean of 6 prompts, lower is closer)

checkpoint	mean LPIPS		checkpoint	mean LPIPS
u500	0.523		u4000	0.488
u1000	0.565		u4500	0.484
u1500	0.553		u5000	0.471
u2000	0.511		u5500 (primary)	0.467
u2500	0.516		u6000	0.477
u3000	0.507		u6500	0.476
u3500	0.493		edit-turbo 4-step	0.485

The trajectory: worst at u1000 (0.565, palette drift), reversed by the v0.2 data fix, monotonically better from u2000, crossing the edit-turbo reference at u4000, bottoming at u5500 before a late plateau. 4-step sampling scores 0.627 — not shippable; a 4-step variant needs a dedicated stage-2. Raw numbers: eval/lpips_sweep.json, full 13-checkpoint grid: eval/grid_sweep.jpg.

3-way eval (same prompts, same seeds) — dev 30-step · edit-turbo 4-step · this LoRA 8-step

Read it row by row: portrait — competitive; ramen — legible main text but a mirrored duplicate above the sign (known failure); alpine valley — comparable; watch — clean product shot (edit-turbo's is warped); library — loss: our staircase dissolves into the bookcase at the top while both references stay coherent; origami whale — full recovery, healthy palette.

LPIPS against a different-composition reference under-reports structural wins: on the product-watch prompt the student is decisively better than edit-turbo (edit-turbo warps the case and melts the crown; the student renders a clean watch, down to an attempted 'CHRONO-21' dial). On in-image text the student still trails both — the v0.1 bank contained zero text_render captions. Methodology + the dev-vs-turbo benchmark that motivated this: akhaliq/fibo2-4step-t2i-benchmark.

Roadmap

• 4-step stage-2: a short critic/DMAD-style run restricted to 4 steps, starting from this 8-step student (the pattern that worked for MOSS-Image)
• ComfyUI: custom node applying the LoRA in fp32 + example workflow
• v0.3.1 nits: residual faint text doubling on complex signs; dial text still fuzzy at small sizes

Usage (diffusers)

import torch
from diffusers import BriaFibo2Pipeline
from peft import LoraConfig, get_peft_model, set_peft_model_state_dict
from safetensors.torch import load_file
from huggingface_hub import hf_hub_download

pipe = BriaFibo2Pipeline.from_pretrained("briaai/Fibo-2.0-dev", dtype=torch.bfloat16).to("cuda")
lora_cfg = LoraConfig(r=128, lora_alpha=128, lora_dropout=0.0, bias="none",
                      target_modules=["to_q", "to_k", "to_v", "to_out.0", "w1", "w2", "w3"])
model = get_peft_model(pipe.transformer, lora_cfg, adapter_name="student")
# IMPORTANT: keep the LoRA fp32 — the distillation delta rounds away in bf16
for n, p in model.named_parameters():
    if "lora_" in n:
        p.data = p.data.float()
set_peft_model_state_dict(model, load_file(hf_hub_download(
    "akhaliq/Fibo-2.0-8step-smd", "update_5500/lora_student_ema.safetensors")), adapter_name="student")
model.set_adapter("student")

# 8-step mu-shifted Euler (mu=1.15 at 1024x1024)
K, mu = 8, 1.15
sigmas = torch.linspace(1.0, 1.0 / K, K, device="cuda")
sigmas = mu * sigmas / (1 + (mu - 1) * sigmas)
sigmas = torch.cat([sigmas, torch.zeros(1, device="cuda")])

prompt = '{"short_description":"an origami whale swimming through paper clouds","objects":[{"description":"large origami whale","texture":"matte folded paper, crisp creases"}],"background_setting":"soft pastel paper sky","lighting":{"conditions":"soft diffused light"},"style_medium":"paper craft diorama","artistic_style":"origami"}'
with torch.no_grad():
    embeds, _ = pipe.encode_prompt(prompt, device="cuda")
    x = torch.randn(1, 32, 128, 128, device="cuda", generator=torch.Generator("cuda").manual_seed(0))
    for j in range(K):
        t = torch.full((1,), sigmas[j].item(), device="cuda")
        v = model(hidden_states=x.to(model.dtype), timestep=t,
                  encoder_hidden_states=embeds, encoder_attention_mask=None, return_dict=False)[0].float()
        x = x + (sigmas[j + 1] - sigmas[j]) * v
    latents = BriaFibo2Pipeline._patchify_latents(x)
    bn_mean = pipe.vae.bn.running_mean.view(1, -1, 1, 1)
    bn_scale = (1 / torch.sqrt(pipe.vae.bn.running_var + pipe.vae.config.batch_norm_eps)).view(1, -1, 1, 1)
    latents = BriaFibo2Pipeline._unpatchify_latents(latents / bn_scale + bn_mean)
    img = pipe.vae.decode(latents.to(pipe.vae.dtype), return_dict=False)[0]
    image = pipe.image_processor.postprocess(img, output_type="pil")[0]
image.save("fibo2_smd_8step.png")

BriaFibo2Pipeline is on diffusers main (pip install git+https://github.com/huggingface/diffusers).

ComfyUI: the LoRA is a plain diffusers-layout LoRA over the diffusers-layout transformer; apply it in fp32 (a stock bf16 merge rounds away most of the distillation delta — measured ~81% relative error on the delta in the analogous MOSS setup). A custom-node wrapper and merged ComfyUI checkpoint are planned.

Lineage and notes

• Teacher/base: briaai/Fibo-2.0-dev (paper) — CC BY-NC 4.0; this LoRA is non-commercial too, commercial use needs a Bria agreement
• Method: SMD (arXiv 2610.18472, code); an earlier data-space DMD attempt (arXiv 2610.12102) is archived at akhaliq/Fibo-2.0-8step-dmd
• Training curves: trackio dashboard
• Motivation: Bria's edit-turbo does 4-step t2i reasonably well for free, but degrades on precision subjects (evidence); this LoRA is trained on the t2i distribution directly