Skip to content

[Bug] Paragraph content lost when heading text overlaps with body text during title-based splitting #6543

Closed

Description

@kelvin-qin

Contact Information

talk about issue

MaxKB Version

v1.10.x ~ v2.x (all versions affected)

Problem Description

When splitting Markdown/PDF documents by titles, if a heading's text also happens to appear as a substring inside the body content (either earlier in the same section or in a later section), the splitter cuts the paragraph at the wrong position, causing part of the real body text to be lost from the resulting paragraph.

This happens because SplitModel.parse_to_tree and get_level_block locate the start/end of each block using text.index(start_content, cursor) / text.index(next_content, ...) — i.e., they search for the first occurrence of the title string in the text, rather than using the actual character offset returned by the regex match that originally found the title. If the title string coincidentally recurs inside the body, index() locks onto the wrong occurrence, mis-defining the block boundaries.


实际发现就是此时,标题与正文内容重叠时,导致分段截断错误,正文丢失、混乱!

Steps to Reproduce

  1. Create a Markdown (or PDF) document where a heading's exact text also appears verbatim inside the body of a preceding or following section, e.g.:
    # Chapter 1  
    This chapter talks about Chapter 1 and Chapter 2 in advance, giving context before diving in.  
    More real content for chapter 1 continues here...  
    
    # Chapter 2  
    Actual content of Chapter 2... some foo bar you know this is not CARE Chapter 1  just title Chapter 2 xxx

2. Upload this document to a MaxKB knowledge base and let it auto-split by title (default pattern-based split).

3. Inspect the resulting paragraph for "Chapter 1" in the paragraph list.

### The expected correct result

The paragraph for "Chapter 1" should contain the full body text: "This chapter talks about Chapter 2 in advance, giving context before diving in. More real content for chapter 1 continues here..."

Actual Result
Because get_level_block finds "Chapter 2" via text.index(next_content, start_index + 1), and the substring "Chapter 2" first appears inside the Chapter 1 body (in "talks about Chapter 2 in advance"), the block is truncated there. The paragraph content ends up being only "This chapter talks about " and the rest of Chapter 1's real content ("giving context before diving in. More real content for chapter 1 continues here...") is silently dropped from the paragraph, moved incorrectly, or lost.



### Related log output

```shell
There is no directly associated log data. This constitutes a quality bug that manifests silently.

Additional Information

Suggested Fix Direction
Instead of re-locating title strings via text.index/text.find after they've already been captured by regex, retain and reuse the actual match.start()/match.end() offsets from re_findall/parse_level when slicing blocks, so that boundary detection is based on position rather than content re-matching.

Activity

  1. kelvin-qin commented on Jul 30, 2026

    @kelvin-qin
    Author

    There is no directly associated log data. This constitutes a quality bug that manifests silently.

  2. Shenguobin0102 commented on Jul 30, 2026

    @Shenguobin0102

    您好,感谢使用,我们无法复现您所描述的问题。

  3. kelvin-qin commented on Jul 30, 2026

    @kelvin-qin
    Author

    @Shenguobin0102 建议重新打开,我把复现文件给你,你们一测遍知。使用你们官网的文档为主体进行测试,通用知识库-上传文档-智能分段,你查看分段结果和原文的对比就能看出问题。

    这个bug从v1版本到v2一直都存在。

    知识库test2.pdf

  4. kelvin-qin commented on Jul 30, 2026

    @kelvin-qin
    Author

    比如原文第一段是:

    Image

    实际分段内容被下个标题“创建知识库”错误分割成这样:

    Image
  5. kelvin-qin commented on Jul 30, 2026

    @kelvin-qin
    Author

    @Shenguobin0102 @zyyfit 使用附件pdf可轻易复现问题。

  6. kelvin-qin commented on Jul 31, 2026

    @kelvin-qin
    Author

    @baixin513 请核实,谢谢,这个bug真实存在为什么直接关了?

  7. baixin513 commented on Aug 3, 2026

    @baixin513
    Member

    感谢反馈,该问题我们已重现,会在后续版本修复。

  8. added this to the v2.10.5 milestone on Aug 3, 2026
  9. baixin513 commented on Aug 6, 2026

    @baixin513
    Member

    v2.10.5已发布

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions