Argos plants a unique canary in every recipient’s copy of your eval set, traces a leak to its source, and checks whether a model has seen it.
Turn protection on for a repo, open a pull request, and read the review.
Recorded from the Argos prototype. The interface text in the recording is in Korean.Eval and data companies share hidden test sets with labs and vendors. Every copy is identical, so nobody can say where a leak came from.
Access logs and contracts can’t tell you which recipient’s copy ended up outside.
Nobody knows which models trained on the eval set. After-the-fact overlap checks miss cases.
At Appen, data was reportedly sold privately before it was published, with no prior disclosure.
Hand over an eval file. Each recipient gets a copy carrying its own canary markers.
Found part of the set outside? Match its markers against the registry to see whose copy it was.
Give a model a canary prefix and check whether it completes the unique suffix.
An illustrative run of the flow we’re building.
Illustrative. The model probe uses a mock model in our demo, and detection power on real models is not yet validated.
Findings land as a comment on the PR, ranked by severity, with a merge recommendation and a short lesson for whoever wrote the code.
API1Object read or modified directly by an ID from the request parametersapp.mjs:45 · broken object-level authorizationAPI7Server makes a request to a user-supplied URL (SSRF)app.mjs:52 · server-side request forgeryAuthentication asks “who are you?”. Object authorization asks “are you allowed to access this data?”. A logged-in user must still not reach another person’s data.
Easy to miss: a login middleware or UUID ids don’t solve ownership. How guessable an ID is and who may access it are separate questions.
Think about it: users A and B are both logged in. What should happen when B requests A’s order ID? If the order isn’t shared by policy, the data must not be returned or changed. The server has to check the relationship between the authenticated user or tenant and the object.
Illustrative PR comment from the static pass. Dynamic attack-and-reproduce checks run separately, and only reproduced findings are reported as verified.
Hand each recipient a distinct copy, protect the system that holds the copies, and know the answer when a leak or a suspect model turns up.
The center of Argos. Per-recipient canaries, a registry, leak identification and model queries.
Attack the system that stores and serves the copies, open a fix, and re-verify with the same attack. Runs in a disposable container.
Hand over a file instead of wiring specs, scenarios and environments. AI drafts the config for you to review.
Based on public product descriptions; we haven’t verified competitor behavior. We found no commercial product offering per-recipient tracing plus contamination detection.