∞Direct answer
Direct answer
Ask for proof behind the startup's AI product score. See which cases trained the model and which guided changes. Check that final test cases stayed out of those steps. If test answers guide changes, the final test is not independent. [1] [2] Ask a named tech reviewer to check fresh cases kept aside. Mark unknown training data as unknown. A high score is not proof of a good investment.
Link each score claim to a test
Due diligence means checks before you invest. This guide checks a startup's AI product, not your research tool. Use it from pre-seed to Series A. Start with the exact claim in its pitch. Ask which user task the score tested.
Build a claim-to-test proof pack: files that support that claim. Save the full task and prompt text. A prompt is the text that guides the model. Add the model version, product code version and test date. Record which source files and tools the product could use.
Name the scorer: the person or tool that judged replies. Save the pass rules and the full count of scored cases. This count is the denominator for a pass rate. Show failed calls, skipped cases and reasons too. Model cards give context on model tests, uses and limits. [4]
- Claim and user task, not just a broad model score.
- Full prompt, model and code versions, files and date.
- Scorer, pass rules, total cases and excluded cases.
Map training, tuning and the final test
Ask the team to name each set of cases. Training cases teach the model. Tuning cases help the team choose model or prompt changes. Final test cases check the chosen version. They must not guide changes to it. Google sets out these roles. [2] Ask for dated records, not just folder names.
Test data leakage means test clues reached model building. Those clues can include the answers. This can make scores look too good on new work. It can be a mistake, not a security breach. Even steps that clean data can let test clues in. [1]
Ask what the team trained or changed itself. Ask what the outside model firm says it trained on. If those training files are not known, write unknown. A check of the team's files cannot prove outside files clean. Keep that limit beside the score.
- Dated records for training, tuning and final test cases.
- Prompt changes, model choices and data-preparation steps.
- Known startup data and unknown outside training contents.
Split the cases to match real work
A split divides cases into sets with different roles. Choose it for the task the startup claims to solve. Google says test cases should reflect real work. It sets no fixed share for each set. [2] A random split is not right for every task.
For a claim about new clients, keep each client's cases together. Do not scatter close copies across training and final tests. This is a group split. It checks work on groups not used in training. scikit-learn explains why linked cases need this care. [3]
For a task that predicts later events, split by time. Build with older data and test on later cases. Do not let future facts enter the inputs. Random splits can mislead on time-linked data, scikit-learn warns. [3] Ask the tech lead why the chosen split fits.
- New-client claim: separate whole client groups where needed.
- Future-event claim: older build data, later test cases.
- Task fit: explain the split and what it cannot test.
Look for reused cases before trusting results
Ask for an overlap check across the case sets. Overlap means the same case appears in more than one set. Check exact copies and close rewrites of the same case. Google warns that training copies in a test make it unfair. [2] Save the check method and which files it could inspect.
Here is one made-up case, not a real startup result. A product test has 100 scored cases. The tech lead finds 20 copies of training cases. That is 20% overlap: 20 divided by 100, times 100. This is a count of copies, not a new pass rate.
Do not delete those results and call the rest clean. That does not show the test stayed apart from tuning. Fix the split and build a new score on fresh cases. Keep those cases out of training and tuning. Note which data the check could not reach.
- Exact copies, close rewrites and shared source records.
- Check method, data covered and data not checked.
- Fresh test required; no patched pass rate.
Run a fresh test with a named reviewer
A holdout is a set kept aside from building and tuning. Ask a named tech reviewer to oversee fresh holdout cases. That person must know the task and how to test it. Use only files the team has rights to test. Keep cases and answers safe with clear access rules.
Freeze the product, prompt and pass rules before the run. Keep the answer key out of the product's inputs. Allow source files only if the real task allows them. Include normal cases, hard cases and missing facts. Set pass rules for this task, not one score for all.
Save each output and its score, not just the best run. Check disputed scores by hand, including scores given by AI. If final results guide changes, those cases become tuning data. They no longer count as an independent final test. Google warns that repeated use can wear test sets out. [2]
- Named reviewer, fresh holdout and protected answer key.
- Frozen product version, prompt, allowed inputs and scorer.
- All outputs, score disputes and any changes after testing.
Keep failures and business proof in view
Ask to see failed cases beside the headline score. Note which tasks and languages were not tested. State what happens when the product cannot answer. Google says model cards should show uses and limits. They should show weak areas too. [4] Do not hide a key fault inside an average.
Keep the claim, dated proof and open gap together. Add the tech lead's name and the next test needed. Missing test records do not prove fraud. Mark the claim as not yet checked. If AI helps read the pack, use only approved files. Keep a person in charge of the judgment.
A sound test backs one product claim within its scope. It does not prove demand, low costs or repeat sales. Check those business claims with their own proof. It does not prove the startup is a good investment. No test score proves future funding or returns.
- Visible failures, untested work and unknown training data.
- Claim, proof, gap, reviewer and next test.
- Separate evidence for demand, costs and sales.
FAQ
Frequently asked questions
Does test data leakage mean a security breach?
Not by itself. Here, test clues or answers reached model building. That can make a score look too good. [1] A breach is a separate question about data access or loss. Check that too if private data may have been exposed.
Can an investor prove an outside model never saw the cases?
Not from a startup's file check alone. The model firm's training files may be unknown. Say so. Fresh cases kept safe help test the current product claim. They do not prove all past training data clean.
What share of data should be kept for final tests?
There is no fixed share for every task. Google sets no percentage rule. [2] Ask the tech lead for enough cases to test the claim. Match real work and keep linked cases apart where needed. A small test leaves more doubt.
REF
Sources
- 1
- 2
- 3
- 4
