OKF: I Couldn't Find a Reason to Adopt It Yet

In my previous post, I introduced a way to organize an organization's knowledge with OKF. This time, I tried it in an AI report for sellers that had been generated from ordinary Markdown. Across 180 runs, I compared three approaches: providing the original documents, summarizing them by concept, and automatically selecting the concepts needed.

With the current volume of documents and the way we use them, I couldn't find a reason to adopt OKF right away in place of ordinary Markdown. Summarization sometimes reduced the input, but I found no advantage in the reports' judgments or preservation of necessary content substantial enough to justify changing the existing approach. Factoring in the operational burden of preprocessing and retrieval, I decided that keeping ordinary Markdown was the better choice for now.

The time to test it again is when the text becomes much longer than it is now. Once feeding in all the documents at once becomes difficult, OKF may help as an index that points to the knowledge needed. That possibility is the next question to test with more documents, rather than a result established by this experiment.

01. Three Inputs, One Pipeline

The report is HTML that reads a seller's product, social media, and marketplace status, then suggests changes in the form of cards. Since a generation pipeline already existed, there was no need to build three copies of it. I swapped only the input at two points: page_sources, which fetches the sources, and page_payload, which builds the request body.

Summary Bundles Went into the DB, Not a Folder of Files

I sent the source documents and the generation prompt for the relevant area to a preprocessing LLM together. I parsed the returned result into a {path, content} array and stored it in a single DB row. Logically, these are multiple concept files, but the implementation does not create an actual file tree.

In one case, creating the initial summary took about 302 seconds. Shorter input therefore does not immediately mean a lower total cost. How much the cache is reused matters. The cache key included the source hash and preprocessing prompt identifier, but changes to the generation prompt were missing from the invalidation conditions. Since the summary is tailored to the generation objective, this also needs to be managed in production.

This implementation takes inspiration from OKF to create summaries by concept. I instructed it to use frontmatter, but did not attach a conformance validator. The experiment also did not include using trust metadata such as verified and stale_after.

Concept Retrieval Became a Separate Subsystem

The retrieval approach splits the source into concepts and loads them into the DB, then reads an index and selects the paths it needs. Documents that need verbatim preservation are stored without conversion. I also built a path for comparing the selections against reference paths and displaying metrics in an admin interface.

The system I builtConcept retrieval → Report generation

The retrieval approach reused the selection cache in all 60 runs. What this experiment repeatedly measured was therefore reports generated from six inputs that had each been selected once, rather than a system performing a fresh search on every run. Because the implementation skips comparison with the reference answers when reusing the cache, retrieval precision and recall were empty in all 60 runs. I measured output repeatability, but could not directly score retrieval at its core job.

02. What the Number 180 Contains

The experiment ran on September 2–3, 2026. I generated each of six requests—products, social media, and marketplaces for two sellers—ten times using each of three input methods. Below, I call a group with the same request and input method a “cell.” There are 18 cells in total.

I fixed the model and reasoning effort and turned off additional refinement and candidate screening. For the original-versus-summary comparison, I alternated the methods within each page. Concept retrieval was added in a separate run, so this was not an experiment in which all three methods were randomly assigned together. In 3 of the 6 retrieval cells, the source documents came from a different point in time than those used by the other methods.

Thus, “180 runs” in this post refers to the size of the experimental design. The stability table uses 173 records; the search for suspicious text uses the 181 records with surviving HTML. One timeout had been omitted by the filename pattern used in the earlier aggregation. That must not be read as evidence that there were no failures.

03. Similar Words, or the Same Content?

The first metric is a mechanically calculated Jaccard similarity of word sets. After stripping HTML tags, I formed word sets, compared every pair of outputs within each cell, and took the average. A cell with ten outputs remaining has 45 pairs. Those pairs share outputs, so they do not represent 45 independent experiments.

I compared sets of card titles in the same way. Word similarity reflects vocabulary repetition, while title similarity reflects part of the layout. Neither directly measures factual accuracy or content completeness. Similarity can be high even if the exact same item is omitted every time.

The second measure is a content review. I removed tags, classes, and image URLs, leaving a “skeleton” of titles, judgments, evidence, checks, and table rows for an LLM reviewer to read. It evaluated five dimensions—judgments, suggestions, numbers, checks, and structure—and extracted items that stayed the same and items that changed. In a separate verification pass, the resulting 134 discrepancies were classified as 96 confirmed, 37 partially confirmed, and 1 unconfirmed. These results were not exhaustively verified against human-authored reference answers.

04. Summaries Led to More Repeated Words

For word similarity, the summary bundle scored higher than the original in all six requests. The cell averages were 62.4 for the original, 72.6 for summaries, and 59.1 for concept retrieval. The average difference between summaries and the original was 10.2 percentage points.

These results mix the effects of input length, summarization, and more uniform phrasing. Since this was not an experiment that changed only the format while keeping the content identical, the increase cannot be isolated as an effect of OKF syntax itself.

Differences between request areas were also substantial. The average was 86.7 for products, 60.1 for social media, and 47.2 for marketplaces. The spread between area averages was 39.6 percentage points, compared with 13.5 percentage points between input-method averages. Across these six requests, the observed differences between request areas were larger than those between input methods.

Similar Words Do Not Guarantee the Content

Average judgment-consistency scores were similar: 90.0 for the original, 89.8 for summaries, and 89.2 for retrieval. Score variation during reevaluation was greater than these differences, making it difficult to use these scores to determine which method produced more stable judgments. The effects I could confirm were less input and more similar wording. Those alone were not enough to conclude that the output had improved sufficiently to justify replacing the current Markdown approach with OKF.

05. Evidence and Structure Changed More Often than the Direction of Recommendations

The item lists extracted by the reviewer were more interesting than the scores. I regrouped by content the 312 items classified as “always the same” and the 216 classified as “different across runs.”

The share classified as fixed was 79.1% for judgments and suggestions, and 43.5% for figures. This does not mean that “the judgment was reproduced eight times out of ten.” These are proportions within item lists with different denominators. The values also change depending on how finely an item is divided.

Even so, this showed where to pay attention when reading the reports. Recommendations could point in a similar direction while the supporting numbers and additional checks changed. Actions were not always fixed either: 4 of the 14 severe discrepancies concerned judgments and suggestions.

06. Other Problems Found During Repeated Runs

Apart from the input-method comparison, defects in generation also stood out. One output had 753 characters and 0 cards, yet was saved as a success. The other nine outputs in the same cell averaged 10,371 characters and all had 6 cards.

Some results also contained strings such as ennials or traces of intermediate work in the middle of sentences. Searching all 181 HTML outputs for infrequent Latin-script tokens produced 23 candidates for another review. All three methods had them: original 5/61, summary 10/60, and retrieval 8/60. Since rare but legitimate words could also be included, I did not treat these figures directly as contamination rates.

These defects need to be addressed separately in the generation pipeline. I changed the way knowledge was delivered in this experiment, but that change alone did not make the output dependable enough to use without concern.

07. Keep Ordinary Markdown for Now

Summarization clearly had a compression effect. In the product area, it reduced input by about 60–65%. But for inputs that were already short, such as social media, it made them larger.

The table shows input character counts. Summarization adds LLM calls and cache management, so fewer characters do not immediately mean lower total cost. Above all, being able to compress the input is not by itself a reason to adopt OKF. The criterion should be whether the benefit is substantial enough to change what is currently done with Markdown.

Concept retrieval did not support making the switch either. Compared against key output items from the other methods, only 18/46 or 25/51 items were preserved, and 12 of the 14 severe discrepancies came from retrieval. These figures are comparisons between output items, rather than recall measured against a retrieval answer set.

The retrieval implementation I built comprised 5,820 lines across production code, tests, and DDL, with 20 routes and 4 tables. That scale reflects the maintenance burden of my implementation, rather than a requirement of the OKF specification itself. Without a clear advantage over the existing approach, there was no reason to add that burden to production.

The conclusion of this experiment is therefore to keep ordinary Markdown in the current service and put OKF adoption on hold. Whether summarization or conceptualization is possible and whether it needs to be adopted now turned out to be separate judgments.

08. Test It Again as an Index When the Documents Get Longer

Currently, I specify and pass along the list of Markdown documents needed for each request. If the number and length of the documents keep growing, managing that list or supplying all the source text at once may become burdensome. At that point, an index that locates the necessary concepts and their source passages could become useful.

What I want to establish is whether an index built with OKF can find all the knowledge needed with less input. As document volume increases in stages, I need to compare passing ordinary Markdown directly, adding a simple table of contents to Markdown, and using an OKF concept index. This would distinguish the effect of having an index from the effect of choosing OKF.

If that comparison shows practical gains in preserving necessary items, input volume, processing time, and cost, adoption can be reconsidered. Because this retrieval experiment repeatedly reused cached selections, it did not establish how well the index could find the right material as the number of documents grows.