How to Choose Software for Detecting AI-Generated Images

Choose AI image detection software by testing whether it improves a defined review decision on the files your team actually receives. The purchase should produce interpretable evidence, a manageable review queue and a clear route for files the service cannot assess.

Use three stages rather than starting with a large benchmark:

1. Desk review: confirm required tasks, file support, data terms and access to the intended product or plan. Eliminate clear mismatches before uploading anything.
2. Authorized usability sample: try a small selection of representative, permitted files to see whether reviewers can understand the outputs and take a useful next step. This is a fit check, not an accuracy estimate.
3. Formal evaluation: test the remaining candidates against independently established labels, agreed error limits and the same workload. Use the rubric and cost comparison below to record the decision.

Use the request for information at the end to put these requirements to shortlisted providers.

Start with the decision the software must support

Write a requirement that names the material, the question and the next action. For example: “Triage product-condition photographs for possible content changes, then send flagged or unresolved cases to a trained reviewer who can request an original file.” This gives a buyer something to test. “Find fake images” does not.

Define the relevant concern before assigning a positive label: permitted AI use and routine editing may be acceptable. Whether a seller is honest or a photographed event occurred as described requires context and corroboration beyond a detector score.

Specify whether you need to detect entirely generated images, identify local alterations, inspect provenance records or combine these tasks. A photograph containing one generated region is a different evaluation case from an entirely generated scene. A genuine photograph used in a misleading transaction might pass both checks while the transaction still needs investigation.

NIST's November 2024 overview distinguishes provenance tracking from synthetic-content detection and explains that individual techniques are context-dependent building blocks rather than comprehensive solutions. That distinction is useful when defining scope. NIST AI 100-4, sections 1 and 3

Record the expected file formats, intake channels, quality range, daily volume and sensitivity. Include ordinary complications such as images downloaded from a messaging platform or supplied as screenshots. Assign an owner to the resulting review queue. If nobody can act on a flag, the product's most impressive capability may have little operational value.

Desk review: separate required capabilities from attractive extras

Ask each vendor to complete the same coverage matrix. Require a dated answer tied to the offered plan and product version. Mark each capability as available, limited, unsupported or planned; a roadmap commitment should not satisfy a current requirement.

For provenance, “credentials found” and “credentials validated” are different statements. Ask which validation checks run and how missing, invalid and untrusted records are represented. The version-pinned C2PA 2.2 explainer describes credentials as tamper-evident provenance information; it does not treat validation as a judgment that the represented event is true. C2PA explainer, goals and non-goals

Evaluate features that address your requirement before scoring extras. A polished heatmap cannot compensate for unsupported input formats, and a large list of supported generators does not itself demonstrate local-edit performance.

Fileverity image verification covers AI generation, image manipulation and provenance in one report, with Content Credentials read and validated against the C2PA specification. Start with the image verification scope, the Trust & security page and pricing, and use your coverage matrix to map those checks to your task. Tell us how your reviewers will use results (web app, export or integration) and the volume you expect, and we will confirm the access method that fits, including API access and batch processing under an Enterprise agreement.

Use gates before a weighted score

Some requirements should disqualify a purchase until resolved. Examples include prohibited use of submitted material for training, an incompatible processing location, unsupported critical file types or no workable route for uncertain cases. Your organization sets these gates. A strong detection score should not cancel a failed data-handling requirement.

After candidates pass the gates, use a weighted rubric to structure comparison. The weights below are a starting point for a review-led workflow; change them before seeing vendor results.

Score each row from zero to four: zero means no evidence; one means a written claim; two means a demonstration or partial documentation; three means evidence your team has checked against its requirements; four means that evidence meets the acceptance condition agreed beforehand. Use pilot results for performance and usability, and reviewed contractual or operational documentation where those are the relevant evidence. Calculate points as weight × score ÷ 4, then sum to a maximum of 100. Record the evidence and caveat beside every score.

The number is a discussion aid. Do not convert missing information into a neutral middle score or choose a universal pass mark. A candidate can achieve a high total while failing a mandatory task. Record that failure explicitly and defer the purchase or narrow the intended scope.

Try an authorized usability sample before a formal pilot

Once the desk review clears the requirements needed to share files, choose a small authorized sample that includes ordinary intake, a known relevant concern and a file that may be unsuitable. Keep sensitive material out until its handling is approved. The aim is to discover obvious workflow mismatches cheaply, not to advertise a success rate from a handful of images.

Ask the intended reviewers to use the actual interface or integration they would receive. Can they identify the input, explain the output, distinguish an inconclusive result from a negative one and write the next evidence request? Record confusion and handling time. If the interface or input support fails this check, resolve that problem before investing in a formal accuracy comparison.

Ask what an accuracy claim actually measured

Request the test population, label definition, threshold, product version and denominators. “Accuracy” can hide whether the sample was mostly photographs or generated images. Ask separately how many relevant positive cases were flagged, how many legitimate cases were flagged, and how many inputs did not receive a usable assessment.

Published research helps illustrate why scope matters. The CIFAKE study evaluates a specific binary classification task using real images and latent-diffusion images reflecting CIFAR-10 classes. Its reported result is evidence about that experiment; it is not a measurement of a commercial service on customer photographs, local edits or unfamiliar workflows. Bird and Lotfi, CIFAKE

For each vendor figure, obtain enough detail to reconstruct the claim: the count of positives and negatives, the counts of errors, and treatment of excluded files. Ask whether related variants of the same source image appeared across development and evaluation data. If a vendor cannot disclose its dataset, request a description sufficient to assess relevance and retain your own held-out pilot as a separate check.

Agree on a common comparison rule. One useful choice is a fixed false-positive-rate budget: compare the recall each candidate achieves while meeting that budget. Another is a fixed recall target: compare the false-positive rate and review workload needed to reach it. Choose the rule before seeing the final results and show uncertainty around the estimates. The same numeric score threshold across different services is not a common operating point because their score scales may differ.

Choose thresholds using development data, then evaluate the frozen settings on the holdout. If a service exposes no threshold control, evaluate its available setting and report whether it meets the chosen constraint; do not imply that all candidates were tested at an identical error rate. The distinction between scores and thresholded decisions, and the need for separate threshold-tuning data, are explained in scikit-learn’s threshold guide.

A score of 0.9 is not automatically a 90% chance that an image is generated. Ask for its definition and evidence for any probability interpretation. The accuracy and false-positive guide explains the denominators, calibration and threshold trade-offs in detail.

Formal evaluation: build a pilot with defensible labels

Use authorized material with documented origins. For photographs, retain capture or acquisition records where available. For generated or edited examples, retain creation records and a description of what changed. Another detector's prediction is not an independent ground-truth label. Put uncertain-origin files into a separate exploratory set instead of forcing them into the main accuracy calculation.

Create two complementary collections. A diagnostic set deliberately includes enough examples of each important task to reveal failures. An operational sample approximates the mix and quality of incoming work so you can estimate queue size and handling time. A balanced diagnostic set should not be used directly to predict live alert frequency when real intake has a different mix.

Separate threshold selection from final evaluation. Keep original images and their related variants within the same partition, then freeze the evaluation files and settings before the comparison. Show initial reviewers the same contextual information across candidates, but keep expected labels hidden until their assessments are recorded. Preserve raw service outputs before discussing them with the vendor.

Hypothetical 1,000-image diagnostic set for a product-photo workflow; not a prescribed sample size or intake mix.

Tag file format, source, subject matter and delivery condition so you can inspect important subgroups. A small subgroup can expose a serious issue but may be too small to estimate its error rate precisely. Zero observed errors should be reported with the denominator, not advertised as proof of a zero error rate.

The technical principle: measure the risk–coverage trade-off

A service can reduce errors among the cases it answers by leaving harder cases unresolved. That may be a sensible design, but it changes how procurement should interpret “accuracy.” The technical framework is selective prediction: evaluate both the share of eligible cases receiving a definite assessment and the error rate within that assessed subset. The two measures are called coverage and selective risk. Geifman and El-Yaniv, SelectiveNet, ICML 2019, section 2

For a labeled evaluation set, let N be the number of eligible cases, A the number receiving a definite assessment and E the incorrect assessments among those A. Then coverage = A / N and selective error = E / A. If A = 0, the latter is undefined, not zero.

The following constructed results illustrate three operating points on the same 1,000-case evaluation set. They assume all requests completed and all cases were eligible, so the unassessed remainder consists only of uncertain cases.

The first point has 99% accuracy among answered cases, but leaves half the inputs unresolved. Moving from the intermediate to broader point adds 150 answers and 33 errors in this hypothetical set. The added group therefore has an error rate of 33 / 150 = 22%, assuming nested assessed subsets. That marginal calculation exposes a trade-off hidden by the headline averages.

Keeping uncertain, failed and inapplicable assessments distinct is one principle used in Fileverity's image analysis. Selective prediction provides an evaluation framework for these outcomes, rather than a description of Fileverity's model architecture. A failed request or unsupported input should be reported separately, not recoded as either an uncertain completed assessment or a negative finding.

For buying decisions, compare vendors at an agreed coverage or error budget and count the remaining review work. Break errors into false positives and false negatives because their costs differ. A lower selective error is valuable only alongside a coverage level and resolution workload your team can accept.

Count unresolved inputs and review work

Keep the result states explicit: flagged, not flagged, inconclusive, unsupported and technical failure. An inconclusive result means the service could not reach the intended assessment; a technical failure means the assessment did not complete. Neither should silently become “authentic.”

Report false-positive rate on eligible known-negative inputs and detection rate on eligible known-positive inputs, specifying whether unresolved cases are excluded. Also report end-to-end detection yield: relevant positives flagged divided by all relevant positives submitted. Together, these measures reveal a service that performs well on a narrow assessed subset but declines much of the workload.

Measure the full review route. Record time spent opening the case, understanding the output, checking supporting evidence, requesting another file and closing or escalating it. Capture disagreement between reviewers. A result that looks precise but repeatedly leads to unsupported conclusions is a usability problem even if the underlying classifier performs well.

In the comparison below, both fictional candidates flag 80 target images and miss 20. They differ in false alarms. Every flag receives one review, and there are no additional unresolved inputs. Here, each “case” in the figure means one image, rather than a multi-image submission.

Hypothetical comparison: 1,000 assessed images, 100 target positives, 900 negatives; eight minutes per flag and assumed labor of US$30/hour. Software fees and missed-case costs are excluded.

Both candidates have the same detection yield of 80%, but B creates 27 extra reviews. At the stated assumptions, that is 216 extra minutes and $108 more review cost per 1,000 images. A lower software price would need to offset that cost before B became cheaper on these two components. Add missed-case, delay and resubmission costs to adapt the comparison to your intake.

Inspect the evidence a reviewer can retain

Ask for representative outputs covering a flag, an ordinary negative result, an inconclusive assessment and a failed request. Confirm that a reviewer can connect each result to the submitted file, processing time and available service-version identifier. Where appropriate, retain a file hash to distinguish exact files; a hash does not establish that the scene is truthful.

If a heatmap appears, ask whether it depicts model attention, a localization prediction or another quantity. Require a legend and a clear explanation of limitations. Do not treat visual emphasis as independent confirmation of an alteration.

Test whether reviewers can write a bounded record: “The service flagged the submitted image for possible generated content. The original file and supporting photographs have been requested. No conclusion about the transaction has been reached.” This separates an observed output, an inference, a corroborating step and a business action.

Verify data handling and operational behavior

Have the appropriate internal owners review processing locations, subprocessors, access controls, retention and training-use terms. Request answers by data category: original uploads, previews, extracted metadata, results, application logs and backups. “We delete files” is incomplete unless it identifies what is deleted, when, and what remains.

If deletion or restricted access is required, make its verification part of acceptance. Obtain written clarification of support access and incident notification.

Exercise the actual integration you intend to use. Test an unsupported file, a timeout, a duplicate submission, a capacity limit and a result delivered after the calling workflow has stopped waiting. Confirm which attempts are billed. If you need an API, confirm its availability under the quoted plan.

Agree how product changes are communicated. A threshold, model or preprocessing change can affect comparability with the pilot. Request version visibility where available and define when you will rerun acceptance checks. Retain a fallback process for outages and a named owner who can suspend automated routing if review volume becomes unmanageable.

Make a scoped purchase decision

Include subscription and usage charges, integration work, manual review, resubmissions and ongoing quality checks in the cost model. Use the same workload for every candidate and show the assumptions. Avoid counting speculative prevented losses as guaranteed savings.

The decision record should state the approved task, excluded uses, gate outcomes, scored evidence, pilot results, unresolved limitations and responsible owners. Define the initial deployment size and the events that trigger reassessment. A narrow, conditional approval can be appropriate when one task is supported and another still lacks evidence.

Treat evaluation as an ongoing responsibility: review the approved use when the model, inputs or operating conditions change. NIST's voluntary AI Risk Management Framework places evaluation within this broader risk-management process. NIST AI RMF overview

What to send us

Send the following to Fileverity and to any other shortlisted provider; we will answer each row for your intake and product version. Fill in the buyer inputs and ask for a dated response tied to the offered plan and product version.

Send the completed request to Fileverity and we will answer each row for your intake: what can be demonstrated now, what needs a pilot on your files and what is outside the offered scope. Keep that response beside the coverage matrix and the current pricing, so your purchase rests on the same workload and requirements you evaluated. Fileverity analyzes files on servers operated by Entogo in Toronto, Canada and deletes them once the analysis is complete. The Trust & security page sets out data handling, including private deployment for Enterprise on request.

By Yang Zheng · Reviewed by Bill Liu