roger pence

Potentially, each day is crucial in the total development

Search
Advanced Search
Folder to search:
Limit search to one of these properties:

    How to create a FlexSearch index

    Install FlexSearch

    Install FlexSearch with this:

    pnpm add flex-search 
    

    or

    node install flex-search
    

    In the root of your project, create a scripts folder and add the following code to your eleventy.js file. The first chunk ensures that the scripts folder is included in your site's build output and the second ensures that the file needed from the FlexSearch package is copied to the scripts folder.

    eleventyConfig.addPassthroughCopy("./src/scripts");
    
    eleventyConfig.addPassthroughCopy({
        "node_modules/flexsearch/dist/flexsearch.bundle.module.min.mjs":
            "scripts/flexsearch.bundle.module.min.js",
    });
    

    Defining and populating a FlexSearch index

    The code presented in this section is steps 1, 2 and 3 from the worker sequence diagram (click "Worker sequence diagram" below for another look at the sequence diagram).

    Worker sequence diagram
    sequenceDiagram autonumber actor User participant client as client<br>(search.js) participant worker as worker<br>(search-worker.js) note over client, worker: index created <br>when page loads client ->> worker: FlexSearch.CREATE_INDEX activate worker create participant index worker ->> index: create index worker->> client: FlexSearch.INDEX_READY note over client, worker: 5K docs indexed in 2/3 of second! deactivate worker User ->> client: send search request client ->> worker: FlexSearch.SEARCH_INDEX activate worker worker ->> index: search index index ->> worker: search results worker ->> client: FlexSearch.SEARCH_RESULTS deactivate worker client ->> User: render search results
    FlexSearch Web worker sequence diagram

    Creating the index data

    The following examples show how to implement FlexSerch with Eleventy. However, the core code and concepts works with many other JS/TS-based frameworks. For this example, let's assume the Eleventy site has just one markdown folder and it is named posts.

    FlexSearch uses Json documents to define and populate its indices. Here is the markdown frontmatter that needs to be indexed for this project (your frontmatter will almost certainly vary from this).

    ---
    title: Bump Version
    description: This Go program makes it easy to... 
    date_created: 2026-02-25
    date_updated: 2026-06-04
    date_published: false
    rank: 0
    pinned: false 
    published: false 
    tags:
      - go
      - git
      - cli
    ---
    

    Markdown example to be indexed

    FlexSearch needs a Json representation like this of the post markdown documents.

    {
      "title": "Bump Version",
      "description": "This Go program makes it easy to...",
      "url": "/posts/bump-version/",
      "tags": ["posts","go","git","cli"],
      "content": "This Go program makes it easy to...",
      "folder" : "post"
    },
    

    Corresponding Json for Markdown example to be indexed

    To create file that contains an array of the objects shown in Figure 1b, create the following Nunjucks (or whatever template language your Eleventy site uses) template named create-search-index.njk in the root of your Eleventy project. The file containing this serialized Json is named search-index.json and is the data folder of the built site (which Eleventy defaults to the _site folder.).

    The template below creates Json objects like the one shown above in

    ---
    permalink: /data/search-index.json
    eleventyExcludeFromCollections: true
    ---
    [
      {%- for post in collections.posts -%}
        {%- if not loop.first -%},{%- endif -%}
        {
            "title": {{ post.data.title | dump | safe }},
            "description": {{ post.data.description | dump | safe }},
            "url": {{ post.url | dump | safe }},
            "tags": {{ post.data.tags | dump | safe }},
            "content": {{ post.templateContent | striptags | trim | dump | safe }},
            "folder": "post"
        }
      {%- endfor -%}
    ]
    

    A Nunjucks template that creates an arry of the objects shown in

    FlexSearch index data needs an id field to uniquely key each record. For the best performance that key value should numeric. The Nunjuck loopIndex variable assigns the unique id value. In this example, an id key is added to each index object with a unique numeric value when the index is created.

    If the data you're indexing already has an id key, give that original key an alias like this:

    {%- if not loop.first -%},{%- endif -%}
        {
            "originalId": post.data.id  
            "title": {{ post.data.title | dump | safe }},
            ...
    
    

    Aliasing an original id key.

    This template runs every time your project builds. The permalink field causes it to create the needed Json array at /data/search-index.json in your build output directory. This file is deployed with your project and read whenever FlexSearch needs to create an index on the posts folder. The eleventyExcludeFromCollections tells eleventy that this a utility file and should not be included in the build step as a file to include in the site.

    Depending on how many many documents you have in the posts folder, create-search-index.njk may take several seconds of build-step time to run.

    The index data created above is a build-time step. The next step to populate the FlexSearch index is a runtime operation and it occurs on every page.

    ✦

    Defining the FlexSearch index

    The are several options to consider when creating a new FlexSearch Document index. However, for core indexing, you only need to use five of them:

    1. indexObjects: Object[] — A Json array where each element represents indexable content in the frontmatter.
    2. store: string[] — the keys that define the search result object. This is the object array you'll iterate to show your users search results.
    3. index: string[] — the keys that define the keys in the index object that you want indexed. Markdown should be stored in the index object as the content key. See the note below about the special-cased id key.
    4. tag: string[] — This has nothing to do with the frontmatter key 'tags'. Rather, the object property keys specified here can be used to constrain the search to these keys. For example, if the tags array contains title then you can, optionally, constrain the search to only the title keys.FlexSerch documents say that you can constain the search to more than one frontmatter key; I can't make that work. Maybe I'm doing something wrong, but I've only be been to successfully constraing a search to a single key.
    5. tokenize: string — a string that defines how FlexSearch tokenizes the search index. There are several options, but generally you'll want to use either forward or strict.

    Let's assume the keys for the Json object that contains our markdown document data looks like this:

    {
        "title": ...,
        "description": ...,
        "url": ...,
        "tags": ...,
        "content": ...,
        "folder": ..., 
    }
    

    and let's also assume that Json object array is named indexObjects.

    This code defines a FlexSearch index for that data:

    const index = new Document({
        document: {
            id: "id",
            store: ["id", "title", "url", "tags", "folder", "description"],
            index: ["title", "url", "content", "tags", "description"],
            tag: [{ field: "folder" }],
        },
        tokenize: "forward",
    });
    
    • id is a specific key name that FlexSearch uses to uniquely identify each index object.
    • store provdes the keys that define the shape of the search results output object. (In this case, the FlexSearch parlance means you are storing that data for use later.)
    • index the data is indexed by keys provided. By default, all of these keys will be searched.
    • tag is an array of {field: <key name>} objects that constain search to those keys. Specifying folder here later enables searching for the search in only the folder key. While this is an array, I can't make more than one key work with this feature.
    • tokenize is a string that specifies use a partial search. Searching for svel finds all tokens that start with svel.

    The code below populates that index. Note how the id key is incremented on each add to ensure a unique identifier for each object.

    let counter = 0;
    for (const post of indexObjects) {
        index.add({
            id: counter++,
            title: post.title,
            description: post.description,
            content: post.content,
            tags: post.tags,
            url: post.url,
            folder: post.folder,
        });
    }
    
    import indexObjects from "./data/flex-search-data.json" with { type: "json" };
    
    function createIndex() {
        index = new Document({
            document: {
                id: "id",
                store: ["id", "title", "url", "tags", "folder", "description"],
                index: ["title", "url", "content", "tags", "description"],
                tag: [{ field: "folder" }],
            },
            tokenize: "forward",
        });
    
        let counter = 0;
        for (const post of indexObjects) {
            index.add({
                id: counter++,
                title: post.title,
                description: post.description,
                content: post.content,
                tags: (post.tags || []).join(","),
                url: post.url,
                folder: post.folder,
            });
        }
    }
    
    import { Document } from "/scripts/flexsearch.bundle.module.min.js";
    import * as FlexSearch from "./flex-search-messages.js";
    
    let index = null;
    let sourceById = new Map();
    
    self.addEventListener("message", async (event) => {
        const { type, data } = event.data;
    
        switch (type) {
            case FlexSearch.CREATE_INDEX: // Received CREATE_INDEX message.
                index = new Document({
                    document: {
                        id: "id",
                        store: data.store,
                        index: data.index,
                        tag: data.tag,
                    },
                    tokenize: data.tokenize,
                });
    
                let counter = 0;
                sourceById = new Map();
    
                for (const obj of data.indexObjects) {
                    const docId = counter++;
    
                    const indexData = {
                        ...Object.fromEntries(
                            data.indexedKeys.map((key) => [key, obj[key]]),
                        ),
                        id: docId,
                    };
    
                    index.add(indexData);
                }
    
                self.postMessage({ type: FlexSearch.INDEX_READY }); // Sent INDEX_READY message.
                break;
    
            case FlexSearch.SEARCH_INDEX:  // Received SEARCH_INDEX message. 
                const results = search(data.searchArgs);
                if (!results || (results && results.length == 0)) {
                    self.postMessage({
                        type: FlexSearch.SEARCH_RESULTS_EMPTY,
                    });
                } else {
                    self.postMessage({
                        type: FlexSearch.SEARCH_RESULTS,
                        data: { results, searchArgs: data.searchArgs },
                    });
                }
                break;
        }
    });
    

    A Nunjucks template that creates an arry of the objects shown in Figure 1b

    export const CREATE_INDEX = "create_index";
    export const INDEX_READY = "index_ready";
    export const SEARCH_INDEX = "search_index";
    export const SEARCH_RESULTS = "search_results";
    export const SEARCH_RESULTS_EMPTY = "search_results_empty";
    

    Constants used by

    Old code below!

    old text below.

    Before the index is populated, it must be defined. An object that defines the indexed is passed to the FlexSearch Document object's constructure. There are many possible field values for this object (see this link for more info), but

    Have you ever been to a restaurant that has a zillion choices available for dinner? FlexSearch is a little like that. It has a slew of configuration knobs, dials, and switches. Some of these options (to my feeble brain) seem to need a degree in search theory to fully understand. I fiddled around with several options and many either didn't immediate affect the search results or immediately did in a negative way. This set of FlexSearch articles specifies the minimal settings that work for me. If you like to tinker, FlexSearch encourages that until the cows come home.

    1. document
      1. id — uniquely identifies a search record. The docs recommend using a unique numeric value.
      2. store — a string array that defines the shape of the search result. It provides the fields from the markdown document frontmatter (shown above in Figure 1a) that you want to show or filter when displaying search results.
      3. index — a string array defines the fields from the markdown document frontmatter whose content are included in the search index. By default, the search applies to all of the fields in the index array, but you optionally constain the search to specific field. For example, searching for a value in only the document's title property.
      4. tag — names an array of field names from the index array to which you can constrain a search. (See the text immediately following this list for a tag property gotcha.) The index defined below sets tag to the folder field. This enables a more finely-grained search that constrains the search to only records with the the folder value specified. This is handy for constraining a search to specific content type (ie, a KB document or a case study).
      5. tag
    2. tokenize: This controls how FlexSearch treats partial searches. Use forward allows partial searches.

    The FlexSearch docs say you can provide an array of field names with the tag property. I couldn't make more than one field work with the tag property. While more than one might be useful, one is enough for many use cases.

    Figure 2 below defines a Document index for the markdown document shown in Figure 1a. index isn't preceded by let or const because index is defined globally in the search worker. We'll the this code in that context at the end of this article.

    index = new Document({
        document: {
            id: "id",
            store: [
                "id",
                "title",
                "url",
                "tags",
                "folder",
                "description",
            ],
            index: ["title", "url", "content", "tags", "description"],
            tag: [{ field: "folder" }],
        },
        tokenize: "forward",
    });
    
    Figure 2. Define a document index for the markdown document in Figure 1a.

    FlexSearch's Document index object provides the ability to have granular search control (ie, search and filter by document properties). There is also an Index object that a little simpler flat file index. Read more about FlexSearch's index types here. In most cases, the Document index is probably the one to use.

    ✦

    Populating the index

    The "Creeating search data" section above explains how the site's markdown documents are converted into a seralized array and saved in the data/search-index.json file. Figure 1b above showed one of the Json objects from that array. It is also shown below for a quick refresher.

    A markdown document's Json object
    {
      "id": 11,
      "title": "Bump Version",
      "description": "This Go program makes it easy to...",
      "url": "/posts/bump-version/",
      "tags": ["posts","go","git","cli"],
      "content": "This Go program makes it easy to...",
      "folder" : "post"
    },
    
    From figure 1b—an element from the index data Json document.

    To initiate defining and populating the search index, this anonymous function is called from search.js after the DOM is loaded. It fetches the Json array and sends the FlexSearch.CREATE_INDEX message to the worker (defined in search-worker.js). The Json array is passed as the data argument for the message.

    // Fetch Json index data and create the index.
    (async () => {
        const indexObjects = await fetch("/data/search-index.json").then((res) =>
            res.json(),
        );
        worker.postMessage({ type: FlexSearch.CREATE_INDEX, data: { indexObjects } });
    })();
    
    Figure 3. Anonymous function to initiate creating the search index

    The worker's message handler receives the FlexSearch.CREATE_INDEX message and creates and populates the index(very quickly). The last thing the handler does is send the FlexSearch.INDEX_READY message.

    Creating and populating the index in context

    import { Document } from "/scripts/flexsearch.bundle.module.min.js";
    import * as FlexSearch from "./flex-search-messages.js";
    
    let index = null;
    
    self.addEventListener("message", async (event) => {
        const { type, data } = event.data;
    
        switch (type) {
            case FlexSearch.CREATE_INDEX: // Received CREATE_INDEX message.
                index = new Document({
                    document: {
                        id: "id",
                        store: [
                            "id",
                            "title",
                            "url",
                            "tags",
                            "folder",
                            "description",
                        ],
                        index: ["title", "url", "content", "tags", "description"],
                        tag: [{ field: "folder" }],
                    },
                    tokenize: "forward",
                });
    
                let counter = 0;
    
                for (const obj of data.indexObjects) {
                    index.add({
                        id: counter++,
                        title: obj.title,
                        description: obj.description,
                        content: obj.content,
                        tags: (obj.tags || []).join(","),
                        url: obj.url,
                        folder: obj.folder,
                    });
                }
    
                self.postMessage({ type: FlexSearch.INDEX_READY }); // Sent INDEX_READY message.
                break;
    
            case FlexSearch.SEARCH_INDEX:
                // This code coming next! 
                break;
        }
    });
    
    Figure 4a. Anonymous function to initiate creating the search index
    export const CREATE_INDEX = "create_index";
    export const INDEX_READY = "index_ready";
    export const SEARCH_INDEX = "search_index";
    export const SEARCH_RESULTS = "search_results";
    
    Figure 4b. JS constants for consistent worker message identification

    Figure 4a explanation:

    Line(s) Explanation
    1 Import FlexSearch Document object
    2 Import messsage ID constants
    4 Declare global (to this file) document index
    6 Listen for messages sent to the worker
    7 Get type and data values from event.data
    (event.data.posts is the Json index data array)
    10 FlexSearch.CREATE_INDEX message received in the worker
    11 - 26 Define and instance Document index object
    28 Initialize id counter
    31-39 Add a post element to the index, incrementing id for each one
    42 Send FlexSearch.INDEX_READY message
    That message notifies search.js that the index is ready
    45 FlexSearch.SEARCH_INDEX message received
    ✦

    Index other folders

    This example creates an index for markdown documents in the posts folder (notice how figure 1c iterates over the collection.posts collection). In most applications, you'll probably want to index more than one folder's markdown documents.

    That's easy to do by adding more for loops in figure 1c to iterate over other folder collections. If you are using a single index (as shown here), to make sure that your other content types all provide the fronmatter fields required by this indexing scheme. They can have other, unique-to-that-collection fields, but they must provide the fields your searching scheme requires.

    If you have content types with varying frontmatter fields, you can also create multiple indexes with FlexSearch. You could have logic that creates the necessary index based on program logic.

    If you had a reviews collection you also wanted to index, the Evently template shown in figure 1c have a second loop added to iterate the reviews collection.

    ---
    permalink: /data/search-index.json
    eleventyExcludeFromCollections: true
    ---
    [
    {% set loopIndex = 0 %}
    
    {%- for post in collections.posts -%}
    {%- if not loop.first -%},{%- endif -%}
    {
        {%set loopIndex = loopIndex + 1 %}
        "id": {{ loopIndex }},
        "title": {{ post.data.title | dump | safe }},
        "description": {{ post.data.description | dump | safe }},
        "url": {{ post.url | dump | safe }},
        "tags": {{ post.data.tags | dump | safe }},
        "content": {{ post.templateContent | striptags | trim | dump | safe }},
        "folder" : "post"
    }
    {%- endfor -%},
    {%- for review in collections.reviews -%}
    {%- if not loop.first -%},{%- endif -%}
    {
        {%set loopIndex = loopIndex + 1%}
        "id": {{ loopIndex }},
        "title": {{ review.data.url | dump | safe }},
        "description": {{ review.data.description | dump | safe }},
        "url": {{ review.data.url | dump | safe }},
        "tags": {{ review.data.tags | dump | safe }},
        "content": {{ review.data.description | striptags | trim | dump | safe }},
        "folder": "reviews"
    }
    {%- endfor -%}
    ]
    
    Figure 5. Including the "posts" and "reviews" in the index data.