图书情报知识 ›› 2026, Vol. 43 ›› Issue (4): 18-27.doi: 10.13366/j.dik.2026.04.018

• 二十大精神研究阐释与践行·高质量数据集建设 • 上一篇    下一篇

词元(Token)经济背景下的高质量数据集建设

丁波涛   

  1. 上海社会科学院信息研究所,上海,200234
  • 出版日期:2026-07-10 发布日期:2026-10-06
  • 通讯作者: 丁波涛(ORCID: 0000-0001-9661-6448),博士,研究员,研究方向:智慧城市、数据治理、数字经济,Email: dbt@sass.org.cn。
  • 基金资助:
    本文系上海市哲社规划“研究阐释党的二十届四中全会精神”专项课题“面向'主权AI'的人工智能治理体系研究”(2026VQH015)和上海社会科学院重大课题“基于隐性知识转化理论的制造业'老师傅'语料开发与治理研究”(2026ZD015)的研究成果之一。

High-Quality Dataset Construction in the Context of the Token Economy

DING Botao   

  1. Institute of Information Science of Shanghai Academy of Social Sciences,Shanghai,200234
  • Online:2026-07-10 Published:2026-10-06
  • Contact: Correspondence should be addressed to DING Botao, Email: dbt@sass.org.cn, ORCID: 0000-0001-9661-6448
  • Supported by:
    This is an outcome of the Special Project "Research on the Governance System of Artificial Intelligence for Sovereign AI"(2026VQH015)affiliated to the project "Interpretation of the Spirit of the Fourth Plenary Session of the 20th Central Committee of the Communist Party of China" supported by a grant from the Philosophy and Social Science Planning Program of Shanghai , and the Major Project "Research on Corpus Development and Governance of Manufacturing 'Veteran Skilled Workers' Based on Tacit Knowledge Transformation Theory"(2026ZD015)supported by Shanghai Academy of Social Sciences.

摘要: [目的/意义]词元经济重塑大模型产业价值核算逻辑,传统以规模优先为核心的数据集建设模式产生显著效能悖论。本文旨在厘清词元经济下高质量数据集建设的目标变革逻辑,构建适配词元效率约束的全流程数据集建设新模式。[研究设计/方法]搭建“目标—模式”分析框架,梳理大模型全生命周期数据需求差异,剖析数据集建设各环节原有制度互补锁定效应,从数据采集、清洗、标注、评估等环节推演全链路转型方案。[结论/发现]高质量数据集建设核心目标由单纯扩充数据体量转向最大化单位词元边际价值,传统数据集各环节分工协同体系因新经济约束失衡,仅局部改良无法消解结构性矛盾,必须开展全生命周期系统性改造;高质量数据集建设需转向精准化采集、语义级清洗、语用级标注、动态化词元效率评估的新型运作体系。[创新/价值]提出以词元消耗、推理收益为核心的动态评估体系,形成覆盖数据生产至模型推理的一体化治理方案,可为政府数据标准制定、企业数据集开发、行业数据协同建设提供对策参考。

关键词: 词元经济, 高质量数据集, 数据产业, 人工智能, 大语言模型, 范式变迁

Abstract: [Purpose/Significance] Token economy has reshaped the value accounting logic of large model industries, and the traditional scale-prioritized dataset construction mode has led to a prominent efficiency paradox. This paper aims to clarify the logic behind the transformation of objective in the construction of high-quality datasets under the token economy, and establish a new full-process dataset construction mode adaptable to token efficiency constraints. [Design/Methodology] This paper constructs an "Objective-Mode" analysis framework, sorts out differentiated data requirements throughout the full lifecycle of large models, analyzes the lock-in effect of original institutional complementarities in each link of dataset construction, and deduces full-link transformation solutions covering data collection, cleaning, annotation and evaluation. [Findings/Conclusion] The core objective of dataset construction has shifted from merely expanding data volume to maximizing the marginal value per token. Under new economic constraints, the original collaborative division system across all dataset development stages have fallen out of balance. Partial incremental enhancements, while beneficial, are insufficient to resolve structural contradictions. Consequently, a systematic transformation of the entire lifecycle is indispensable. Dataset construction needs to shift to a new operation system featuring targeted collection, semantic-level cleaning, pragmatic annotation, and dynamic token efficiency evaluation. [Originality/Value] This paper proposes a dynamic evaluation system centered on token consumption and inference benefits, and develops an integrated governance scheme covering the whole chain from data production to model inference. It can provide countermeasure references for government data standard formulation, enterprises datasets development, and industry data collaborative construction.

Keywords: Token economy, High-Quality datasets, Data industry, Artificial Intelligence, Large Language Models, Paradigm shift