The knowledge base acts as a pipeline, and then files are uploaded through the API. Since the files do not follow the pipeline process, they cannot be recognized. What's the problem here?

[Richie666888]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Content

Using version 1.9.0, the JSON of Body is as follows:

{"indexing_technique":"high_quality","process_rule":{"rules":{"pre_processing_rules":[{"id":"remove_extra_spaces","enabled":true},{"id":"remove_urls_emails","enabled":false}],"segmentation":{"separator":"\n\n","max_tokens":1024},"parent_mode":"full-doc","subchunk_segmentation":{"separator":"\n","max_tokens":512}},"mode":"hierarchical"},"doc_form":"hierarchical_model","doc_language":"English","retrieval_model":{"search_method":"hybrid_search","reranking_enable":true,"reranking_mode":"reranking_model","reranking_model":{"reranking_provider_name":"langgenius/openai_api_compatible/openai_api_compatible","reranking_model_name":"Qwen/Qwen3-Reranker-8B"},"weights":null,"top_k":3,"score_threshold_enabled":true,"score_threshold":0.5},"embedding_model":"Qwen/Qwen3-Embedding-8B","embedding_model_provider":"langgenius/openai_api_compatible/openai_api_compatible"}

[Richie666888]

@crazywoola

[dosubot[bot]]

It looks like files uploaded via the API aren't being recognized by the pipeline because the API upload process and the pipeline workflow are not fully unified in Dify 1.9.0. When you upload a file through the API, it triggers document processing, but it may not execute all custom pipeline steps (like custom extractors) as the UI does. In practice, the API often defaults to basic parsing and chunking, so some pipeline features might not run as expected unless you trigger them separately. There is currently no single API call that both uploads a file and runs the full pipeline in one step—you need to upload the file, then trigger the pipeline run with a dedicated API call if you want the full pipeline to execute. This is a known limitation and there is ongoing discussion about adding a unified "create-by-pipeline" API endpoint to streamline this process (details).

If your files are stuck in "queued" or "embedding waiting" status, double-check that all middleware services (PostgreSQL, Redis, vector DB), Celery workers, and embedding/model configs are healthy and consistent. For hierarchical (parent-child) structure, make sure your API payload uses "process_rule": { "mode": "hierarchical", ... } and "parent_mode": "paragraph" (not "full-doc"), as using "custom" mode or the wrong parent_mode can collapse the structure to a single chunk (reference).

If you recently upgraded Dify or are using datasets created in older versions, schema mismatches (like doc_form differences) can also cause files not to be recognized—creating a new dataset usually resolves this (reference).

In summary: for full pipeline processing via API, upload the file, then trigger the pipeline run with the appropriate endpoint and parameters. If you need hierarchical chunking, ensure your API payload matches the dataset's configuration and uses the correct process_rule settings.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}