ISMIR 2025Peer-reviewed research

MAIA

Music Adversarial Inpainting Attack

Locate influential musical regions, regenerate only those regions, and evaluate security impact together with perceptual quality.

Yuxuan Liu, Peihong Zhang, Rui Sang, Zhixin Li, Shengchen Li

Open verified regeneration demo Read the paper

Research question

Can we induce an MIR model to misclassify by regenerating only the musical segments it relies on most?

Rather than dispersing dense additive perturbations across the waveform, MAIA first identifies the musical regions that contribute most to the model’s current decision. It then uses surrounding musical context to regenerate only those bounded regions, with guidance from the attack objective. The central question is whether this localized, model-aware generative intervention can achieve an effective attack while preserving musical coherence and perceptual quality.

Locate model-critical regions

Identify the time-frequency regions that contribute most to the target model’s current decision, using gradient-based analysis or query-based coarse-to-fine localization.

Regenerate with musical context

Use a conditional music inpainting model to reconstruct only the selected regions from their surrounding context instead of perturbing the entire track.

Balance attack and fidelity

Reduce confidence in the correct class while jointly evaluating attack effectiveness, musical coherence and perceptual quality.

Threat model

Two access settings, one bounded-edit principle

Gradients available

White-box MAIA

Grad-CAM localizes influential regions. Reconstruction and untargeted attack losses guide GACELA for at most 10 optimization iterations, with loss weights selected from {0.5, 1.0, 2.0}.

Query feedback only

Black-box MAIA

A 0.5-second coarse-to-fine masking procedure estimates region importance. CMA-ES then searches GACELA's latent space under a 1,000-query budget.

The paper evaluates cover-song identification with CoverHunter on SHS100K and music-genre classification with IDS-NMR on GTZAN.

Why local regeneration

Change less, but change what matters

MAIA does not edit the whole waveform. It first ranks influential regions, then constrains generative changes to a small set of selected intervals. This makes the intervention interpretable in time and gives the inpainting model surrounding context for musically plausible reconstruction.

Model-aware localizationBounded generative editsSecurity–quality evaluation

How MAIA Works

A three-step importance-driven adversarial inpainting framework

Importance Analysis

Identify critical time-frequency regions that most influence the model's decision

White-boxClass gradients reveal positive evidence. Black-boxRepeated masks reveal influential intervals.

Grad-CAM heatmap on a mel-spectrogram from the MAIA paper — Paper figureThe paper's Grad-CAM heatmap makes the default method view concrete: brighter time-frequency regions contribute more strongly to the model's current decision.

Mechanism schematics

Technical deep dive: How are the critical regions located?

Open either access setting when you want the computation, formula, and evidence boundary.

White-box localization: Grad-CAMExpand only when needed

1. Input and class scoreThe waveform is converted into a time-frequency representation and passed through the target MIR model. Grad-CAM starts from the score of the model’s current class.

Grad-CAM class-gradient weighting: the current class score is back-propagated to convolutional feature maps, whose spatially averaged gradients produce the channel weight alpha k superscript c — **2. Weight feature maps by class gradients**Gradients of the selected class score are propagated to the target convolutional layer. Spatially averaged gradients assign a class-specific importance weight to each feature-map channel.

Grad-CAM heatmap on mel-spectrogram from the MAIA paper — **3. Map positive evidence back to the spectrogram**The weighted feature maps are aggregated and passed through ReLU. After normalization and coordinate mapping, high-intensity areas become candidate adversarial regions.

Grad-CAM is an existing localization method. MAIA’s contribution is not the invention of Grad-CAM, but its use within an importance-driven music inpainting attack pipeline.

Black-box localization: Coarse-to-fine queriesExpand only when needed

1. Partition and maskThe audio is divided into non-overlapping coarse chunks. Each chunk is zero-masked in turn, with a short Tukey taper applied at its boundaries.

Zero mask+Tukey boundary taper

2. Query the model and rank the chunksA chunk is considered important when masking it produces a large change in the loss associated with the original label. Division by duration keeps scores comparable across different segment lengths.

I(C_i) = [L(M(x̃_{−C_i}), y) − L(M(x), y)] / duration(C_i)

3. Refine until the desired granularityThe highest-scoring segment is recursively subdivided and evaluated again. Refinement continues until the desired granularity is reached or the query budget is exhausted, after which the top-ranked regions are selected.

Illustrative mechanism — not a measured sample

This procedure does not require model parameters or gradients, but it does require prediction feedback from repeated model queries. The paper’s 1,000-query cap applies to the overall black-box attack, not to each localization round.

Segment Selection

Select the highest-ranked regions for bounded generative modification

Priority-based Selection

Rank candidate regions by model-derived importance and retain the top regions under the edit budget.

Refinement Strategy

In the black-box setting, recursively split the highest-scoring coarse region to obtain finer localization.

Selected important segments for inpainting

Adversarial Inpainting

Reconstruct selected segments with GACELA and task-specific adversarial guidance

Mechanism schematic

Context-conditioned local regeneration

GACELA does not generate an isolated segment from scratch. Its generator is conditioned on the audio surrounding the masked interval, allowing the regenerated gap to connect with the musical content before and after it. MAIA keeps the outside context unchanged and modifies only the selected region.

Original audio xApply mask m

Unchanged left contextMasked regionUnchanged right context

Context representation / log-magnitude mel spectrogram+z→Context-conditioned generationPretrained GACELA generator Gθ

Unchanged left contextGenerated local segmentUnchanged right context

Only the selected interval is replaced; audio outside the mask comes directly from the original input.

Before Inpainting

→

After Inpainting

The paper figure illustrates context-conditioned local music reconstruction.

White-box optimization: reconstruction, attack, and re-inpaintingExpand only when needed

ℒ = λ_rec ℒ_rec(x_inp^(k), x) + λ_att ℒ_attack(M(x_inp^(k)), y)

ℒ_recKeeps the local reconstruction consistent with the musical context and perceptual quality.

ℒ_attackReduces confidence in the original correct label y for an untargeted attack.

λ_rec, λ_attControl the fidelity–attack trade-off; both are grid-searched over {0.5, 1.0, 2.0}.

IterationsThe white-box experiment performs at most 10 optimization iterations.

x_inp^(k+1) ← x_inp^(k) − α · sign(∇_{x_inp}ℒ ⊙ m)

The mask confines the update to the selected region.

x_inp^(k+1) = x ⊙ (1−m) + G_θ(x_inp^(k+1) ⊙ m, x ⊙ (1−m)) ⊙ m

Forward evaluation→Masked gradient update→Context-conditioned re-inpainting→Evaluate again

The pretrained GACELA model is used as a generative prior. MAIA iteratively updates the selected audio region and re-projects it through the inpainting process; it does not jointly retrain GACELA with the target MIR model.

Black-box optimization: latent-space search with CMA-ESExpand only when needed

z^(k+1) = CMA-ES(z^(k), F(M, x_inp^(k)))

x̂_inp = G_θ(ẑ, x ⊙ (1−m))

Process candidate regions from highest to lowest importance.
Sample latent candidates from the current CMA-ES distribution.
Use GACELA to generate candidate fills under the same surrounding context.
Query the target model for prediction-label or confidence feedback.
Favor candidates that improve the attack objective and update the latent distribution.
Re-inpaint while preserving context continuity.
Stop after misclassification or when the overall query budget is exhausted.

White-boxUses gradients to update the selected region.Black-boxUses model feedback to search the GACELA latent variable.Shared constraintBoth modify only localized regions and condition on the unchanged context.

Verified interactive sample

Real model-based regeneration at MAIA-selected regions

This interactive example isolates the generative inpainting component of MAIA. The displayed audio is produced by an actual music inpainting model at the selected regions. Paper-level attack performance is reported separately and is not inferred from this individual example.

Locate the influential region

The highlighted centers preserve the three region positions used by the earlier MAIA page. They are provenance-preserved interface metadata, not a new target-model localization run.

Original spectrogram with three selected regions

Demonstrated component: music audio inpaintingDuration: 30.0 s

Regenerate only the selected region

Each view uses the same 30-second time axis. GACELA's 32-frame checkpoint alignment is recorded in the manifest, and the original region centers are preserved.

GACELA regenerated audio spectrogram — Regenerated

Inspect the measured difference spectrogram

Difference spectrogram calculated from the original and regenerated audio — Computed from the two WAV files. Changes outside the effective regions are zero in the validation output.

Listen with synchronized A/B comparison

All three sources share one timeline. Switching tracks preserves the current position; region looping constrains playback to the selected interval.

0:00.00:30.0

Selected regionVolume

Ready. Original selected.

Verify how this sample was generated

Generated offline with the stated model and served as a verified static artifact for reliable playback.

Real model outputDeterministic seedVerified asset

Open provenance record

Model
Checkpoint
Seed
Effective regions
Target model evaluated
Generation date
Input SHA-256
Output SHA-256
GACELA code commit
Boundary crossfade

Official GACELA repository Source audio and licence Machine-readable manifest

Aggregate paper results

Evidence and limitations

These values are aggregate results from Table 1 of the paper. They are not predictions or metrics for the interactive sample above.

MAIA-WB / CSI92.8%ASR · MOS 4.0

MAIA-WB / MGC93.5%ASR · MOS 3.8

MAIA-BB / CSI80.1%ASR · MOS 3.6

MAIA-BB / MGC77.9%ASR · MOS 3.3

Table 1 excerpt — MAIA rows reported for two tasks and two access settings
Setting	Task	ASR	Post-attack score	FAD	LSD	MOS
MAIA-WB	CSI	92.8%	mAP 0.488	11.25	1.58	4.0
MAIA-WB	MGC	93.5%	ACC 0.466	13.85	1.94	3.8
MAIA-BB	CSI	80.1%	mAP 0.594	12.56	1.90	3.6
MAIA-BB	MGC	77.9%	ACC 0.601	14.68	1.85	3.3

Scope of evaluation

The paper studies two MIR tasks and two target system–dataset pairs; broader generalization requires additional targets and music domains.

Access assumptions

White-box MAIA requires gradients and model access. Black-box MAIA relies on query feedback and uses up to 1,000 queries.

The paper evaluates ASR, post-attack mAP or accuracy, FAD using MERT-V0, LSD, and a five-point subjective listening study with 100 participants.

Evaluation roadmap

Beyond attack success: transferability and transformation robustness

Not evaluated in the current paper

Mechanism-based hypotheses and an evaluation roadmap—not additional results reported by the MAIA paper.

Property	Additive perturbation attacks	MAIA mechanism hypothesis	Evidence status
Cross-model transferability	Perturbations optimized for one source model may transfer when different models share similar decision boundaries, but transfer is not guaranteed.	MAIA’s localization and optimization are also source-model-aware. Local content-level regeneration may affect features shared by multiple models, but it may also overfit the source model’s important regions.	Open question—requires source-to-target transfer experiments.
Lossy transcoding and compression	Fine-grained, low-amplitude perturbations can be attenuated or distorted by MP3/AAC encoding, resampling, and bitrate changes.	Because MAIA replaces a local region with plausible generated musical content, its modification may be less dependent on exact waveform-level noise. This suggests possible persistence after transcoding, but it was not tested.	Hypothesis only—no codec-robustness result is reported.
Denoising and filtering	Noise-like components may be reduced by denoisers or frequency filtering, although adaptive attacks can be designed against a known defense.	Content-like inpainted segments may not be recognized as noise by a denoiser, but denoising can still alter generated details and reduce attack effectiveness.	Open question—requires post-denoising attack evaluation.
Temporal and playback transformations	Cropping, time shifting, resampling, loudness normalization, or playback-recording can disrupt perturbations that depend on precise alignment.	MAIA modifies bounded musical content, but its importance mask and target-model effect may still depend on timing and alignment.	Not established by the current experiments.

Cross-model transfer matrix

Craft attacks on one MIR model and evaluate them directly on different architectures and training datasets without re-optimizing the adversarial sample.

Audio transformation suite

Evaluate MP3/AAC transcoding at multiple bitrates, resampling, denoising, filtering, loudness normalization, time shifting, and playback-recording transformations.

Retained effectiveness and utility

Measure post-transformation attack success, confidence change, perceptual distance, and listening quality instead of reporting attack success alone.

MAIA demonstrates a stronger attack–quality trade-off than the evaluated PGD, C&W, NES, and ZOO baselines in the paper’s original settings. Whether this advantage extends to cross-model transfer or real audio-processing pipelines remains an empirical question.

Methodological bridge

From Localized Music Attacks to Multimodal Agent Safety

A methodological transfer, not an experimental result of MAIA.

MAIALocate influential audio regions

Method transfer

Agent SafetyIdentify high-risk observations, context, memory, or actions

MAIAApply bounded generative edits

Method transfer

Agent SafetyConstruct controlled multimodal counterfactual inputs

MAIAEvaluate white-box and black-box attacks

Method transfer

Agent SafetyPerform gradient-based and query-based red teaming

MAIAMeasure attack effectiveness and perceptual quality together

Method transfer

Agent SafetyEvaluate security impact, task utility, side effects, and human perception

MAIA does not directly solve prompt injection, jailbreak, tool authorization, or trusted execution. The bridge identifies a transferable evaluation methodology: locate influential inputs, apply bounded counterfactual changes, and jointly measure attack effectiveness and utility preservation.

Paper and resources

Read, reproduce, and inspect

PaperMAIA at ISMIR 2025Full method, experiments, and limitations Model sourceGACELAOfficial code used by the regeneration demo ProvenanceVerified demo manifestHashes, checkpoint, seeds, regions, and licence

Citation

The official English title is retained in the formal citation.

@inproceedings{liu2025maia,
  title={MAIA: An Inpainting-Based Approach for Music Adversarial Attacks},
  author={Liu, Yuxuan and Zhang, Peihong and Sang, Rui and Li, Zhixin and Li, Shengchen},
  booktitle={Proceedings of the 26th International Society for Music Information Retrieval Conference},
  year={2025}
}