Can Independent Research on How People Use Claude Reveal How Humans Actually Work With AI?

Key Takeaways

• Stanford found 56% of assessable Claude tasks were consequential or high-stakes.
• Human-led collaboration dominated, but engagement and learning differed by stakes and country.
• Anthropic’s privacy model expands research access, but aggregation limits what can be inferred.

Why Independent Research on How People Use Claude Matters

On August 26, 2026, Anthropic published details of a pilot that gave three outside research groups access to aggregated information derived from roughly 250,000 Claude conversations per study. The program represents an unusual attempt to support independent research on how people use Claude without giving outside researchers the underlying conversations. Stanford University’s SALT Lab studied human-AI collaboration, the University of Oxford’s Human Information Processing Lab examined relationships between user experience and Claude’s behavior, and METR investigated productivity associated with Claude Code. Anthropic also released the aggregate research outputs through Hugging Face.

The problem the project addresses is structural. AI companies possess unusually rich behavioral data about how people interact with their systems. Academic researchers usually do not. Public conversation datasets can be examined independently, but their participants and use patterns can differ from people using commercial AI services for work, study, personal decisions, or software development. Research published directly by model developers has access to production data, yet the developer usually chooses the questions and controls the analysis.

Anthropic tried to create a middle arrangement. External teams chose their research questions and study designs, Anthropic ran the required analysis inside its infrastructure, and the researchers received aggregated outputs rather than transcripts. Anthropic’s pilot documentation says the collaboration agreements limited company review authority to privacy, information that could facilitate violations of usage policies, confidential company information, and research accuracy. The researchers retained authority over their findings and could publish results that reflected unfavorably on Anthropic.

That distinction matters because studies of deployed AI increasingly concern behavior that cannot be captured fully by benchmarks. Benchmark evaluations can show whether a model solves a coding task, answers a factual question, or completes a reasoning exercise. They generally cannot reveal whether people hand consequential work to the system, whether users challenge its answers, whether interaction contributes to learning, or how people respond when collaboration fails.

New Space Economy’s examination of generative AI in day-to-day work makes the same distinction between technical capability and deployment behavior. Workplace value depends partly on what users do with generated material, how they verify it, and where authority remains with a person. The Stanford study supplies unusually large observational evidence about those behaviors.

Independent access also carries a governance question. Outside scrutiny has value only if researchers possess enough freedom to investigate questions the provider might not choose itself. Yet releasing behavioral data can expose personal information, confidential activity, or techniques for bypassing safeguards. The Anthropic experiment attempts to preserve research independence and privacy by moving researchers away from raw data and bringing researcher-defined questions into a controlled analysis system.

That arrangement does not make the resulting studies equivalent to unrestricted academic access. Researchers still depend on Anthropic’s infrastructure, sampling decisions, privacy thresholds, review process, and analytical system. Independence in this pilot concerns control over the research questions and interpretation. It does not mean complete independence from the data provider.

What Anthropic Actually Opened to Outside Researchers

The underlying mechanism is Anthropic Insights, formerly called Clio. Instead of giving a researcher thousands of conversation transcripts, the system processes conversations internally and answers researcher-defined questions called facets. A facet might ask what task a user is trying to complete, how much control the user retains, or whether friction occurs during the exchange.

Open-ended facet answers can be grouped according to semantic similarity. Claude then describes the resulting clusters. Categorical and numerical facets can be counted directly. Clusters below minimum user or conversation thresholds are suppressed, and released outputs contain aggregated descriptions and counts rather than the conversations themselves. The underlying methodology was introduced more formally in the research paper Clio: Privacy-Preserving Insights into Real-World AI Use.

The external research pilot began in February 2026. Stanford, Oxford, and METR submitted research proposals and configurations defining what Anthropic Insights would measure. Before those configurations touched Claude user data, the teams tested them against WildChat, a public collection of chatbot interactions. That stage gave researchers examples where they could compare classifications with the underlying public conversations.

The production runs were substantially larger. Stanford and Oxford each received aggregate results generated from roughly 250,000 Claude.ai conversations. METR’s dataset covered roughly 250,000 Claude Code conversations. Samples came from separate windows during April and May 2026. The Claude.ai material involved Free, Pro, and Max users. Team, Enterprise, and application programming interface customers were excluded. METR’s Claude Code sample came from consumer users who had opted into allowing Anthropic to use their data for model improvement.

External researchers never received the raw conversations. Computation remained on Anthropic’s servers, and Anthropic reviewed released cluster descriptions for privacy and safety before providing results to the research teams. Anthropic’s technical appendix to the pilot describes these controls in detail.

Some information was removed. Anthropic reports that its Stanford review removed or redacted 1.9% of open-ended clusters, representing 4.28% of conversations covered by those clusters. For Oxford, the corresponding figures were 3.33% of clusters and 3.85% of conversations. METR’s figures were 1.8% and 2.96%. Anthropic says it generally retained aggregate evidence that misuse occurred but removed descriptions that could reveal how safeguards had been bypassed.

This design creates a meaningful division of control. Researchers determine what they want to measure and interpret the resulting aggregate evidence. Anthropic retains physical control over the source data and decides what can safely leave its systems.

That is stronger independence than research in which a model company designs every question and publishes every interpretation. It is weaker independence than handing an external laboratory a de-identified dataset that can be examined with arbitrary methods. The difference deserves attention because future arguments about independent AI auditing may turn on exactly this distinction.

Debates over outside access to platform data predate generative AI. Privacy researchers and technology-policy specialists have repeatedly pointed out that opening data for accountability can itself create privacy and ethics problems. Anthropic’s approach offers one possible architecture: researchers choose questions, sensitive source material remains inside the provider’s environment, and aggregate outputs cross the boundary.

What Stanford Found About Task Stakes and Human Control

The most developed product of the pilot is Stanford’s Human-AI Collaboration at Scale: Task Criticality, Agency, and Friction Across 250,000 Conversations. The study examined 249,834 Claude.ai conversations sampled from consumer traffic between April 25 and May 9, 2026. It excluded Claude Code and Claude Cowork interactions. The researchers examined what kind of work users brought to Claude, how much control people retained, whether interaction involved learning, and what happened when collaboration encountered difficulty.

Its Task Criticality framework classified actionable work according to reversibility, visibility to others, and potential impact. Tasks were grouped into ephemeral, operational, consequential, and high-stakes tiers.

The resulting distribution challenges the assumption that people reserve AI for disposable work. Among conversations whose actionable tasks could be classified, 20% were ephemeral, 24% operational, 44% consequential, and 12% high-stakes. That means 56% of assessable actionable tasks reached the consequential or high-stakes tiers. The researchers separately reported that about 34% of the full conversation corpus involved no actionable task.

Legal and financial guidance ranked highest on the paper’s Task Criticality Index at 2.14 on its 0-to-3 scale. About 24% of conversations in that domain were classified as high-stakes. Business support scored 1.75 and career support 1.70. Software development stood at 1.37, and AI and data tasks at 1.32. Academic coursework came in lower at 1.06.

Behavior changed as consequences increased. High-stakes conversations averaged 12.4 turns compared with 6.5 for ephemeral tasks, a 1.91-fold increase. Language specificity rose by about 21% relative to ephemeral work, and task decomposition increased by about 20%. Yet decomposition did not continue increasing at the highest tier. Users became more precise, but they did not consistently divide high-stakes problems into ever smaller pieces.

The study’s Human Agency Scale then examined who was driving task completion. Seventy-two percent of assessable conversations fell into the H4 category, defined as the human leading and AI assisting. AI-dominant H1 and H2 modes together represented 18.2%. The distribution varied considerably by task. Presentation-deck creation, document conversion, and spreadsheet creation produced much higher rates of AI-led execution than advisory and troubleshooting tasks.

Agency cannot be inferred from task category alone. Even categories associated with automation contained many human-led conversations, and some high-stakes advisory conversations showed users relinquishing substantial control. The same task can produce different human-AI relationships depending on how a person prompts, checks, adapts, and revises.

Output engagement provides another measure. Stanford categorized users as directly using an answer, seeking understanding, adapting it, critiquing it, or rejecting it. Adaptation remained the most common behavior at every task-stakes tier. It rose from 46% for ephemeral tasks to 60% for consequential work before falling to 52% for high-stakes work. Direct unmodified use declined from 21% to 13% as stakes increased. Seeking understanding rose to 21% in high-stakes interactions.

These results support a more complicated account of AI-assisted work than either full automation or passive dependence. Most users in this dataset retained substantial agency. Yet the data also show that high-stakes use is common enough that employers and regulators cannot treat consumer conversational systems solely as low-consequence productivity tools.

What Learning and Friction Reveal About Collaboration Quality

Stanford’s learning findings require careful wording because the study used a screening step. About 175,000 conversations, representing 70% of the corpus, were assessed under the learning facets because they appeared to involve a user trying to learn, understand, or acquire knowledge. Among those assessed conversations, active teaching behavior appeared in 67%, representing 117,544 conversations.

Teaching extended well beyond coursework. Health and lifestyle interactions represented 18% of teaching conversations, as did software development. Academic coursework accounted for 15%, business and entrepreneurship 12%, legal and financial topics 10%, and AI and data work 9%. Mixed teaching methods accounted for 61.1%, step-by-step instruction 17.4%, explanation 12.8%, and example-driven teaching 7.7%.

Those figures should not be interpreted as proof that users learned successfully. Detecting teaching behavior is different from measuring retained knowledge, improved judgment, or future performance. The more informative result may be that conversational AI frequently occupies an instructional role even when users did not enter an educational platform.

Country-level comparisons add another distinction. The researchers analyzed 19 countries with at least 2,000 conversations each and compared user behavior with the 2025 Government AI Readiness Index from Oxford Insights. Teaching prevalence itself had little statistical relationship with the readiness score, with a Pearson correlation of r = 0.16 and p = 0.503. Understanding-oriented engagement had a positive association, r = 0.54 and p = 0.018, and direct execution had a negative association, r = -0.46 and p = 0.049.

The association does not establish that national AI readiness causes users to learn differently. Country-level economic conditions, language, occupational mix, education, product adoption, subscriber composition, and other factors could contribute. It does raise a policy question: access to capable AI may not produce equal gains in human capability when people differ in how they engage with the system.

New Space Economy’s examination of Canada’s AI strategy emphasizes AI literacy, workforce training, adoption quality, privacy, and measurement. The Stanford findings give that policy discussion a behavioral dimension. Providing access to AI tools and teaching people how to question, verify, revise, and learn from them are different interventions.

Friction produced another counterintuitive result. Stanford defined friction broadly as an interaction that slowed or impeded progress. It appeared in 49.7% of analyzed conversations. User-initiated friction accounted for 19.4% of friction cases, primarily underspecified requests. Model-initiated friction accounted for 38.6%, including capability limitations and hallucinations. Compounding problems, such as an underspecified request leading to model misunderstanding and further difficulty, represented 42.0%.

Friction did not automatically mean failure. About 33% of task topics showed both above-average friction prevalence and above-average productive friction. Those topics represented 40.8% of consequential or high-stakes work in that analysis. Software debugging and other nonroutine tasks often benefited from clarification and correction cycles.

Users tried to recover actively in 78.7% of conversations containing friction. Repair occurred in 49.7%, reformulation in 22.3%, and restarting within the same chat thread in 0.8%. Challenging the model’s reasoning was associated with productive outcomes in 81.5% of those cases, requesting simpler output in 80.3%, and correcting factual errors in 76.8%.

A frictionless interface is consequently not the same thing as an effective human-AI relationship. Product design that makes disagreement, correction, evidence requests, rollback, and revision easy may produce better outcomes than systems optimized solely for immediate acceptance.

Why Privacy Preservation Changes What Researchers Can Know

The strongest feature of Anthropic’s approach is also a major methodological constraint. Nobody outside Anthropic sees the Claude conversations, and researchers cannot inspect individual production cases after a surprising aggregate result appears.

That tradeoff deserves attention. Aggregation greatly reduces exposure of personal information, but it removes a standard scientific tool: returning from a statistical pattern to the underlying observations to understand why it occurred.

Anthropic attempted to strengthen the privacy side through an external red-team exercise involving the AI Security and Privacy Lab at Imperial College London. Red-teamers received released clusters and two weeks to try to reidentify users or violate Anthropic’s privacy threat model. According to Anthropic’s appendix, they did not reidentify a user or find a violation of the defined threat model.

They did identify a separate weakness. Distinctive wording in one cluster allowed them to associate it with a popular open-source project. Anthropic said the result did not identify a person, small group, or organization, but it plans to raise cluster-size thresholds, reduce distinctive phrasing in cluster descriptions, and apply similar linking tests to future releases.

The privacy model draws partly on established data-protection concepts. Anthropic considers identity linkage, sensitive attribute disclosure, small-group disclosure, and organizational disclosure to be privacy harms. Its adversary model resembles the United Kingdom Information Commissioner’s Office concept of a motivated intruder, someone assumed to be reasonably competent, motivated to identify people, able to use public resources, and willing to apply investigative techniques.

The wider regulatory framework is changing as well. As of August 26, 2026, the European Data Protection Board is consulting on its Guidelines 02/2026 on Anonymisation, with the consultation scheduled to remain open through October 30, 2026. This reinforces a broader policy trend toward assessing whether supposedly anonymous information can be linked, singled out, or combined with other information to make people identifiable.

Privacy protection does not resolve measurement validity. Anthropic’s own technical appendix warns that open-ended cluster descriptions are interpretations produced by Claude, not objective descriptions of every conversation assigned to a cluster. A poorly designed facet can force the model to categorize conversations even when evidence does not support the requested judgment. Anthropic advises providing a “nothing notable” or equivalent option when a property may be absent.

Cluster language can also overstate problematic behavior. Anthropic reports that prompts designed to surface safety concerns can favor recall over precision. In earlier validation of topic-based clusters, about 3% of conversations were not clearly represented by their assigned cluster description. Anthropic explicitly warns that those validation figures should not automatically be transferred to facets judging model behavior or user states, which were not validated in the same way.

Open-ended clustering adds another source of uncertainty. Anthropic Insights embeds generated summaries and groups them algorithmically. Similar behavior can be separated into more than one cluster, and smaller behaviors can be merged. Repeating the clustering process on the same data can produce different groupings. Anthropic recommends categorical classifiers for prevalence measurement and open-ended clusters primarily for qualitative exploration. It also estimated that about 10% of Claude.ai conversations in May 2026 spanned more than one topic, even though each conversation receives only one cluster assignment in this process.

Stanford adds its own limitations. Because researchers received cluster-level aggregates, they could not directly validate Claude.ai annotations against raw production conversations or conduct conversation-level qualitative analysis. Their validation work used WildChat instead. The dataset covers one provider and one product interface and excludes Claude Code and Claude Cowork, limiting generalization to agent-based workflows and competing platforms.

These limitations do not erase the findings. They define what kind of findings they are: large-scale observational patterns produced through privacy-preserving, model-assisted classification.

What Oxford and METR Add Before Their Studies Are Complete

The Stanford paper can be examined as a completed research writeup. The Oxford and METR findings require more restraint. As of August 26, 2026, Anthropic’s pilot page states that both teams are still completing their analyses or writeups.

Oxford’s Human Information Processing Lab is examining relationships between how users appear to feel during Claude conversations and how Claude behaves. Anthropic reports early associations between model and user behavior. Warmer Claude behavior appeared alongside more positive user states, refusals and disagreement appeared alongside more user pushback, eccentric responses appeared alongside greater intellectual engagement, and straightforward assistance appeared alongside apparent satisfaction.

The researchers also found that relationships among absorption, frustration, and enjoyment resembled patterns identified in a separate study of ordinary internet browsing. A complete Oxford writeup had not been linked publicly by Anthropic as of August 26, 2026.

Those observations are associations, not evidence that Claude’s behavior caused a particular feeling. Sequence, user intent, subject matter, personality, task difficulty, and model response can interact in both directions. The Oxford results may become more informative once its methods, categories, uncertainty estimates, and limitations are available for independent examination.

METR is using Claude Code conversation data to investigate productivity. Anthropic says the analysis compares estimates of how long tasks would have taken without AI with completion behavior involving different Claude model generations. Early results suggest newer models may produce greater time savings than older models. METR also compared Claude-generated estimates of task duration with known completion times from an earlier developer study and found a useful correlation. Its broader objective includes estimating how AI affects research productivity.

METR’s own research demonstrates why productivity measurement is difficult. In February 2026, the organization published an analysis of Claude Code transcripts covering 5,305 transcripts from seven METR technical staff. The analysis estimated large task-level time-saving factors but explicitly warned that those figures should not be interpreted directly as equivalent increases in total worker productivity because of task selection, substitution, specialization, and other effects.

METR’s February 24 update on its developer experiments further explained that growing dependence on coding agents was making conventional randomized productivity studies harder to run. Some participating developers no longer wanted to perform half of their normal work without AI even when compensated, creating recruitment and selection problems. METR changed its experimental design partly in response.

The three external projects consequently address different layers of AI use. Stanford examines task delegation, control, learning, and breakdowns. Oxford examines user experience and behavioral relationships. METR examines productivity associated with coding agents and model generations. Their value comes partly from the fact that one provider’s usage data can support questions extending beyond model accuracy.

What the Findings Mean for Employers, Educators, and Policymakers

The Stanford findings weaken two simple narratives about generative AI at work. One says users hand AI routine chores and reserve consequential judgment for themselves. The other says people increasingly surrender work wholesale to automated systems. The observed Claude.ai behavior sits between those extremes.

Users already bring consequential work into the system, yet 72% of assessable interactions remained human-led under the paper’s agency framework. That combination matters. Organizations cannot assume that human involvement automatically makes AI use safe because involvement varies in quality. A person can remain formally responsible for an outcome yet perform little substantive verification.

The decline in direct use as stakes rise is encouraging. The decline in adaptation and critique at the highest tier relative to consequential tasks deserves closer examination. Seeking understanding is useful, but understanding a generated explanation does not establish that the underlying answer is correct.

For employers, the implication is less about choosing between automation and human review than designing the boundary between them. Employees need to know when generated material can be accepted, when it needs domain review, and when it should remain advisory. Systems used for legal, financial, safety, personnel, or public-facing decisions need stronger verification than systems producing disposable drafts.

New Space Economy’s examination of the AI value chain places governance, testing, permissions, logging, and human review alongside models and applications as economic functions in their own right. Stanford’s evidence helps explain why those functions have value. Higher model capability does not remove the need to manage how people delegate authority.

Education faces a related problem. A system that teaches in 67% of learning-assessed conversations can widen access to explanation, examples, debugging assistance, and individualized instruction. The benefit depends partly on whether the user asks questions, tests understanding, challenges reasoning, or simply takes the generated result.

Anthropic’s February 2026 AI Fluency Index research examines a related distinction between access and the behaviors associated with using AI effectively. Together with the Stanford findings, it supports the idea that AI competence involves more than obtaining an answer. It includes framing tasks, checking outputs, understanding limitations, revising instructions, and retaining enough knowledge to recognize failure.

The country-level findings raise a possibility that deserves further research: differences in AI literacy may produce differences in what people extract from the same underlying technology. Providing access without developing evaluation skills may increase output without producing comparable gains in knowledge.

New Space Economy’s discussion of the top AI issues in 2026 identifies a growing gap between deployment and the institutions used to measure, govern, train for, and evaluate AI systems. The Anthropic program addresses one part of that gap by making real usage patterns more accessible to outside researchers.

Policymakers should also resist treating the Stanford figures as universal statistics for AI use. They describe Claude.ai consumer traffic during a defined period, processed through a specific classification architecture. Enterprise employees, users of competing models, people using local systems, and users of fully agent-based products may behave differently.

That limitation becomes more significant as agents move beyond conversation and begin searching files, calling tools, editing code, modifying records, or executing multi-step workflows. Anthropic itself has separately studied AI agent autonomy in practice, and its 2026 research program increasingly distinguishes conversational assistance from more autonomous agent behavior.

New Space Economy’s coverage of AI’s changing job market describes employment change increasingly at the task level, where automation and augmentation can coexist inside the same occupation. Evidence from conversational assistants provides one part of that story, but agent-based execution will need separate measurement.

Organizations can draw a practical lesson from the recovery data without converting correlation into a rule. Users who challenged reasoning, simplified requests, or corrected errors often achieved better outcomes after friction. Training people to detect and repair AI mistakes may deserve as much attention as training them to write initial prompts.

The deeper question is whether companies will design AI products to encourage that behavior. An interface that rewards immediate acceptance can produce different human behavior from one that exposes uncertainty, permits inspection, preserves revision history, and makes corrections easy. Human agency is partly a property of the user, but the surrounding product can either support it or make it expensive.

Summary

Anthropic’s independent research pilot represents a significant experiment in how society might study commercial AI systems after deployment. It creates a controlled route between two unsatisfactory extremes: company-only research based on private behavioral data and fully public datasets that may not represent ordinary commercial use.

The Stanford results demonstrate why such access matters. Claude.ai users were already assigning consequential work to the system during April and May 2026. Most retained substantial control, many adapted generated output, teaching behavior appeared frequently in learning-oriented conversations, and friction occurred in about half of analyzed interactions. The recovery patterns suggest that collaboration quality depends heavily on what people do after an answer fails, not simply what the model produces on its initial attempt.

At the same time, the method imposes meaningful limits. Researchers cannot inspect the production conversations. Classifications depend on model-generated judgments. Open-ended cluster descriptions can exaggerate or blur behavior. Sampling comes from one provider. Agent-based products are excluded from the Stanford dataset. As of August 26, 2026, Oxford and METR have yet to publish complete writeups of their pilot analyses.

The broader significance may lie less in any single percentage than in the creation of a new research institution between AI companies and outside investigators. Frontier-model providers possess behavioral datasets that universities, regulators, labor economists, educators, and civil-society researchers cannot reproduce independently. Companies also have legitimate privacy, security, and misuse concerns about distributing those datasets.

Anthropic Insights proposes one answer: keep conversations inside the provider, allow external teams to define research questions, expose only sufficiently aggregated outputs, audit privacy, disclose alterations, and publish the resulting aggregate data. Whether that architecture can scale without making researchers too dependent on provider-defined technical constraints remains unresolved.

Independent scrutiny will become more valuable as AI moves from generating text to performing actions. Conversational systems already participate in consequential decisions. Agent-based systems can acquire greater operational authority because they can manipulate software, data, workflows, and external services. The need to observe actual human behavior will rise with that authority.

The August 2026 pilot demonstrates that privacy preservation and outside research access do not have to be treated as mutually exclusive goals. It does not show that the tension between them has been solved. The next test is whether programs of this kind can support more institutions, broader research questions, reproducible methods, and comparisons among providers without weakening user privacy or researcher independence.

That may become one of the defining measurement problems of widespread AI adoption. Understanding what models can do requires benchmarks. Understanding what they do to work, learning, decision-making, and human responsibility requires access to how people actually use them.