# Deviation log: The AI Answer Evidence Index Dated entries after the registration. Registered files are never edited; what happens after the registration is written here and published with the results. Entries are added, never changed. ## Registration - **5 October 2026, 13:32:08 UTC.** The preregistration was published on Zenodo: doi:10.5281/zenodo.23145556 (concept doi:10.5281/zenodo.23145555), version 1.0.0. The owner gave the instruction to publish on the same day, after the method note had been revised following the review round, the second read of the fixes and the rehearsal block. SHA-256 of `MANIFEST.sha256`: `6146ee82ce380fb9350958d8bf90b5e1079afe8f87726ea2577a29ccab509c16`. The record was opened without a login afterwards: the three files match the local copies, and every file of the package matches the manifest. - No collection for the main run had been made before this time. ## Deviations 1. **5 October 2026, main run, block 1: the exit check at the end of the block failed because one of the two IP echo services did not answer.** - At 17:22:27 (+03:00) ipinfo.io answered "HTTP Error 503: Service Unavailable". ipwho.is reported 135.136.3.113, New York, US: the address both services had reported at the block's three earlier checks (16:33:46, 16:53:18, 17:07:32) and reported again at 17:27:59. The VPN client showed the same New York connection throughout. Every check is in `raw/exit-checks.jsonl`. - By the run plan a failed check ends the block. It failed after the block's last question, so no question was left unasked. - The answers saved after the last passing check are Gemini's eight answers of block 1: BIZ-04, TRV-05, AIT-04, FIN-01, FIN-02, ELC-07, AIT-17 and TRV-06, saved from 17:08:00 to 17:22:25. They keep their records. As the run plan says, the results are given with and without them. 2. **5 October 2026, main run: block 4 was run on the first day of the window, not the second.** - The run plan puts blocks 1 to 3 on the first day and blocks 4 to 6 on the second. Block 4 was started at 22:52 (+03:00) on 5 October, two hours and fifty minutes after block 3 had ended. The reason is the calendar: the owner wants the collection finished sooner. - Nothing else changed: the block's questions, their order, the engines' order, the waits, the exit checks and the commands are those of the run plan. The time of every asking is in its record. 3. **6 October 2026: an added analysis and a fourth engine, both outside the registration.** - The owner instructed that the collected answers also be used for a second analysis that the registration does not contain: which sources the engines cite, and how far the engines share them. It is the third round of the Cross-Engine Citation Study. The instruction was given on 6 October, after six of the seven blocks had been collected, before any figure had been extracted and before any count of sources by engine had been made. The analysis is exploratory and is reported apart from the registered measures. - For this added analysis only, the owner instructed that Perplexity be collected as a fourth engine. The registered scripts do not collect it (method note, section 2), so the route is a different one: the study's AI assistant opens `perplexity.ai/search?q=` in a browser that is not logged in, one question at a time, in the registered asking order, from the same New York exit, and saves the answer text and the source list as the page shows them. An AI model therefore takes part in this collection, which the registered collection excludes. - At a verification page the assistant does nothing. The operator may pass it by hand. If the page stays, or at a usage limit, the collection stops and is taken up again later. Every stop is recorded. - The records are kept in `addendum/perplexity/`, apart from `raw/`, and no registered script reads them. Perplexity's answers are not part of the registered measures: no figure of theirs is extracted, checked or classified under the registration, and every registered result is reported for the three registered engines. 4. **6 October 2026, main run: block 7 and the re-ask pass were started on the second day of the window, not the third.** - The run plan puts block 7 and the re-ask pass on the third day. They were started at 22:22 (+03:00) on 6 October, after block 6 had ended at 15:33 and its pages had been fetched by 15:46. The reason is the operator: Google's verification page can only be passed by hand, three blocks had already lost their Google questions to it, and the operator was at the machine that evening. - Nothing else changed: the questions, their order, the engines' order, the waits, the exit checks and the commands are those of the run plan. The time of every asking is in its record. 5. **6 October 2026: a fifth engine for the added analysis, outside the registration.** - For the added analysis of deviation 3 only, the owner instructed that Claude be collected as a fifth engine. Claude is not in the registered run because it is reached through its API, a different access route from the public interfaces (method note, section 2). That route is used here: the study's AI assistant sends each of the 50 questions once to the Anthropic API (Message Batches) with the API's web search tool, an approximate location of New York, NY, United States, no system prompt and the API's default settings. An AI model therefore takes part in this collection, which the registered collection excludes. - The answers are what the model returns to a bare question with a search tool. They are not a reading of the consumer interface, and they are reported apart, with the access route named. - The search cap was planned at 5 searches per question and set to 3 before sending, to keep the worst-case cost estimate within the owner's spending limit of 10 US dollars. A lower cap may reduce the number of sources Claude cites. No answer ran more than one search. - The records are kept in `addendum/claude/`, apart from `raw/`, and no registered script reads them. Claude's answers are not part of the registered measures. 6. **7 and 8 October 2026: the figure chain run on the answers of the two added engines, outside the registration.** - After the registered results of the main set had been computed (7 October, 05:22 +03:00), the owner instructed that the four registered steps (extraction, check, classifier, summary) also be run on the 100 answers of Perplexity and Claude that were collected under deviations 3 and 5. The analysis is exploratory and is reported apart from the registered measures. No registered result changes. - The frozen functions are used as they are, with the same two local models. A script of this analysis writes the records those functions read from the collection records in `addendum/perplexity/` and `addendum/claude/`, and a wrapper points the functions at the folder `addendum/figures/`, which holds the scripts, the outputs and a README. - The pages these answers cite were fetched from 7 October, 23:57 to 8 October, 04:56 (+03:00), 25 to 34 hours after the answers. 147 of the 722 addresses reuse the fetch made for the registered sets. The page fetch required both IP echo services to report New York, United States; the exit's address changes from request to request, so the two reported addresses may differ. - Deviation 3 says that no figure of Perplexity's answers is extracted, checked or classified under the registration. That remains so: this analysis is outside the registration. - No figure of the two engines is in the validation sample. 7. **8 October 2026: the results of the five engines are shown in the same tables.** - Entries 3, 5 and 6 say that the two added engines are reported apart. On the owner's instruction of 8 October the results text and its tables show the five engines together. - The values of the three engines named in the preregistration are unchanged and stand in the same tables. The method names which engines the preregistration held, and Claude is named with its access route (API) wherever it appears. ## Run record Not deviations: what each block did, for the reader of the data. - **Block 1, 5 October 2026, 16:33 to 17:27 (+03:00).** 24 askings, 23 complete. Google showed its verification page at the first question (28 s, passed by the operator) and at no other. Google's attempt at ELC-07 did not finish within the waiting time and is recorded as incomplete; the question is asked again in the re-ask pass. Every address the saved Google pages did not hold was read. The block's cited pages were fetched from 17:28 to 18:06: 80 addresses, 72 readable. - **Block 2, 5 October 2026, 18:07 to 18:42 (+03:00).** Google showed its verification page at the block's first question. No operator was present, the page stayed for the five minutes, and Google was stopped for the block as the method note says: one stopped attempt (SVC-08), and the block's other seven questions were not opened on Google. ChatGPT and Gemini: 16 askings, 16 complete. All four exit checks passed. Google's eight questions of this block are asked in the re-ask pass. The block's cited pages were fetched from 18:42 to 19:02: 34 new addresses; 114 addresses so far, 103 readable. - **Block 3, first start, 5 October 2026, 19:02 to 19:09 (+03:00).** Google showed its verification page at the block's first question (AIT-12); it stayed for the five minutes and Google was stopped for the block. The exit check before ChatGPT then failed: the machine's internet connection had dropped and neither IP echo service could be reached. The block ended there, as the run plan says. No answer was saved between the last passing check (19:02:34) and the failed one (19:09:53), so there is nothing to report with and without. The connection came back within minutes and the exit was New York again. - **Block 3, second start, 5 October 2026, 19:14 to 20:02 (+03:00).** 24 askings, 22 complete. Google showed its verification page at the first question (passed by the operator) and answered all eight questions. Two ChatGPT attempts did not reach chatgpt.com (a navigation timeout at FIN-09, the browser's "site can't be reached" page at BIZ-05); they are recorded as an error and an incomplete attempt, and both questions are asked again in the re-ask pass. Gemini: eight of eight. All four exit checks passed. The block's cited pages were fetched from 20:03 to 20:41: 83 new addresses; 197 addresses so far, 171 readable. - **Block 4, 5 October 2026, 22:52 to 23:26 (+03:00).** Run a day early: see deviation 2. Google showed its verification page at the block's first question (AIT-03). It was not passed within the five minutes, and Google was stopped for the block as the method note says: one stopped attempt, and the block's other seven questions were not opened on Google. ChatGPT and Gemini: 16 askings, 16 complete. All four exit checks passed (New York, 149.102.226.121). Google's eight questions of this block are asked in the re-ask pass. The block's cited pages were fetched from 23:27 to 23:40: 28 new addresses; 225 addresses so far, of which 203 gave readable text in at least one of the two fetches (plain or rendered). Counted the same way, the figure after block 3 is 177; the entry above gave 171. - **Block 5, 6 October 2026, 13:16 to 13:59 (+03:00).** 24 askings, 24 complete: Google eight of eight, ChatGPT eight of eight, Gemini eight of eight. Google's first question (SVC-07) showed a verification page for 12 s. It cleared inside the script's 20-second grace time, so no operator wait was started and the record does not count it as passed by the operator. No other verification page was shown. All four exit checks passed (New York, 149.102.226.104). Every address the saved Google pages did not hold was read. The block's cited pages were fetched from 14:05 to 14:56: 99 new addresses; 324 addresses so far, of which 289 gave readable text in at least one of the two fetches. - **Block 6, 6 October 2026, 14:57 to 15:33 (+03:00).** Google showed its verification page at the block's first question (ELC-10). It was not passed within the five minutes, and Google was stopped for the block as the method note says: one stopped attempt, and the block's other seven questions were not opened on Google. ChatGPT and Gemini: 16 askings, 16 complete. All four exit checks passed (New York, 149.102.226.104). Google's eight questions of this block are asked in the re-ask pass. The block's cited pages were fetched from 15:34 to 15:46: 20 new addresses; 344 addresses so far, of which 308 gave readable text in at least one of the two fetches. - **Perplexity, added collection (deviation 3), 6 October 2026, 17:07 to 22:08 (+03:00).** Not part of the registered run. 50 questions, each asked once in the registered order, through the operator's own Chrome, signed in by the operator to a free Perplexity account; 50 complete answers, 608 source entries. After the operator had passed one verification page by hand before the first question, no verification page, sign-in wall or usage limit was shown. The collection was interrupted from about 18:40 to 21:19, when the desktop app was closed and the network exit dropped; it resumed from the same New York exit service. Question 33 (SVC-07) had been sent at 18:38:57, seven seconds after a passing exit check; its answer was read at 21:21 from the account's stored thread and the question was not asked again. Whether that answer was written before the exit dropped is not on record, so the added analysis is given with and without it. Nine exit checks, all New York, US. A notice that a preview of the advanced search was switched on was seen at questions 1, 2, 3, 43 and 44; at questions 33 to 42 it could not be observed reliably. Every saved answer text reproduces the length and SHA-256 that the page reported (`verify_records.py`: 50 records, 0 failed). Interface language Turkish, answers in English. Records and log: `addendum/perplexity/`. - **Block 7, 6 October 2026, 22:27 to 22:34 (+03:00).** Run a day early: see deviation 4. 6 askings, 6 complete: Google two of two, ChatGPT two of two, Gemini two of two. No verification page was shown. All four exit checks passed (New York, 135.136.3.118). With this block every one of the 50 questions has been put to the three engines at least once. The block's cited pages were fetched from 22:35 to 22:47: 20 new addresses; 364 addresses so far, of which 325 gave readable text in at least one of the two fetches. - **Claude, added collection (deviation 5), 6 October 2026, 22:39 to 22:44 (+03:00).** Not part of the registered run. 50 questions, each asked once in one batch (`msgbatch_01P3wPGr7XsQ3JAGZpJdRsk8`) through the Anthropic API: model `claude-sonnet-5-5` as named by every response, the question as the only message, no system prompt, the API's web search tool called directly with at most 3 searches and New York, NY, United States as its location setting. No exit check applies, because the search runs on Anthropic's servers. 50 complete answers, no error, no second batch. 48 answers ran one search each and cite 3 to 9 pages: 272 source entries, 236 distinct addresses, 190 distinct hosts. Two answers, TRV-08 and TRV-16, ran no search and cite nothing: the model made no search call for them, and no search call of the batch returned an error. No answer reached the search cap or the token limit. Cost from the usage fields: 1.38 US dollars (661,712 input tokens, 47,852 output tokens, 48 searches). Every record matches the raw results file (`verify_records.py`: 50 records, 0 failed). Records, raw results, script and log: `addendum/claude/`. - **Re-ask pass, 6 October 2026, 22:51 to 7 October 2026, 00:02 (+03:00).** Started a day early: see deviation 4. Every block was run once more, in order, and the collector asked only the questions that had no complete record and had attempts left. 27 askings, 25 complete: Google 23 of 25, ChatGPT two of two (FIN-09 and BIZ-05, both at their second attempt). Blocks 5 and 7 had nothing left and ended at once. Google showed one verification page, at ELC-14 (22 s, passed by the operator). Two Google attempts ended as errors of the collector and are saved as such: at ELC-07 (second attempt) the browser session was lost before the page was read, and at ELC-16 (first attempt) the screenshot call did not answer within 90 s. Every exit check passed (New York, 135.136.3.118). Every address the saved Google pages did not hold was read. - **Second pass, 7 October 2026, 00:03 to 00:10 (+03:00).** Two questions were still open, both on Google. ELC-16 was complete at its second attempt. ELC-07 did not finish within the waiting time at its third attempt; its attempts are used up and the question has no Google answer. After the pass the main set holds 149 complete answers of 150: Google 49 of 50, ChatGPT 50 of 50, Gemini 50 of 50. Every exit check passed (New York, 135.136.3.118). - **Cited pages after the re-ask pass, 7 October 2026, 00:10 to 01:06 (+03:00).** The pages cited by the answers of both passes were fetched in one run after the second pass: 115 new addresses; 483 addresses in the main set, of which 429 gave readable text in at least one of the two fetches. The fetch of block 7's pages ended at 22:54, not at 22:47 as the entry above says; counted at its end, the figure after block 7 is 368 addresses, of which 328 were readable. - **Repeat run, block 1, 7 October 2026, 00:25 to 01:15 (+03:00).** 24 askings, 22 complete: Google six of eight, ChatGPT eight of eight, Gemini eight of eight. No verification page was shown. Two Google attempts are recorded as incomplete: at TRV-05 the attempt ended on Google's home page after 22 s, and at ELC-07 the answer did not finish within the waiting time. All four exit checks passed (New York, 135.136.3.118). Every address the saved Google pages did not hold was read. - **Repeat run, block 2, 7 October 2026, 01:16 to 01:24 (+03:00).** 6 askings, 6 complete: Google two of two, ChatGPT two of two, Gemini two of two. No verification page was shown. All four exit checks passed (New York, 135.136.3.118). The repeat run holds 28 complete answers of 30: Google 8 of 10, ChatGPT 10 of 10, Gemini 10 of 10. The cited pages of both blocks were fetched in one run from 01:25 to 02:00: 74 addresses, of which 64 gave readable text in at least one of the two fetches. - **Repeat run, remaining cited pages, 7 October 2026, 14:00 to 14:12 (+03:00).** The 28 addresses of the repeat set that the fetch of 01:25 had not reached were fetched. The repeat set has 102 addresses, of which 92 gave readable text in at least one of the two fetches. The check and the classifier of the repeat set ran after this fetch, from 14:15 to 14:20. - **Figure chain on the added engines (deviation 6), 7 October 2026, 23:41 to 8 October 2026, 05:07 (+03:00).** Not part of the registered run. 100 answers of Perplexity and Claude. Extraction from 23:57 to 02:01, no failure. Pages: 575 addresses fetched from 23:57 to 04:56 and 147 reused; 601 of the 722 addresses gave readable text. Check at 04:56, classifier from 04:56 to 05:07, summary at 05:07. 755 figures in Perplexity's answers and 939 in Claude's; no figure was left unlabelled.