Feeds

Each of your searchable type must have a separate feed in XML, JSON or CSV format. Before you start implementing the feeds, establish what data you will index and figure out the structure. Read the Data layout docs for guidelines.

Product feeds

Product feed is the source of data about your products and the quality of the data in the product feed will have the biggest impact on the search quality. There are 2 aspects of a product feed: structure and contents.

Structure

We are very flexible regarding feed structure and support any valid XML or CSV file. However, to make the processing easier for us (and reduce the time to integration), please adhere to following guidelines:

• We prefer XML over CSV.
• If using XML, avoid using tag attributes. Instead of <product id="123">... use <product><id>123</id></product>
• If using XML, keep the structure as flat as possible. Nesting tags is ok, but don't use complex nesting if possible. In many cases, nesting is unavoidable and it's ok.
• Name the attributes in a way that makes sense to you, there's no prescribed naming convention.

We frequently encounter these common problems when dealing with XML files, if you can avoid these, it will make the process much faster:

• No encoding of special characters, e.g. &. Make sure that all special characters are encoded as entities, or escaped via CDATA directive.<name>Black & White</name> is not a valid XML. <name>Black &amp; White</name> or <name><![CDATA[Black & White]]></name> is valid.
• Double encoding of special character using both CDATA directive and entities. E.g., <name><![CDATA[Black &amp; White]]></name> is syntatically valid, but the CDATA directive will prevent special handling of XML entities, and the text will be parsed verbatim, in this case Black & White. Use either CDATA or entities, not both,

Contents

Make sure that the feed contains all products that you have, even those that are not in stock. For product attributes, it's useful to think about four aspects of the data in the feed:

• Required attributes: We only require two attributes for each product: title and URL. They must be present in the feed for each product (the tag names do not matter) as the bare minimum.
• Visual attributes: Look at the product tile in the category listing and all the information that it shows. If you want the product tile in search results to show the same information, it must be present in the feed. This usually includes product image, product price, discounts, various labels. Keep in mind that some information is only visible under certain circumstances, so it's best to check the code that renders the product tile and make sure that all information is included in the feed.
• Behavioral attributes: These attributes are not required to render the product tile, but they are essential for some features related to the product tile. For example, if your product tile shows an "Add to cart" button, and this button will trigger an XHR request which sends the internal product ID, make sure that this internal ID is present in the feed.
• Searchable attributes: All attributes that your product should be findable by. E.g., if you want your products to be searchable by EAN, you must include product EAN code in the feed. In this regard: more is more, and the more data you provide in the feed, the better the search will be.

Luigi's Box product XML feed

Example of XML feed if you are not using product variants

<?xml version="1.0" encoding="UTF-8"?>
<items>
  <item>
    <identity>NH757A</identity>
    <title><![CDATA[Black Nike Shirt]]></title>
    <url>https://example.org/2172-black-nike-shirt</url>
    <availability>1</availability>
    <availability_rank>3</availability_rank>
    <availability_rank_text><![CDATA[In external warehouse / Ships within 5 days]]></availability_rank_text>
    <category primary="true"><![CDATA[Apparel | Men | Shirts]]></category>
    <!-- If the product is in multiple categories, feel free to add each category+hierarchy as separate tag. -->
    <!-- Make sure that the primary category is marked with primary="true" attribute -->
    <category><![CDATA[Seasons | Summer | Shirts]]></category>
    <brand><![CDATA[Nike]]></brand>
    <price>$78.9</price>
    <price_old>$81.9</price_old>
    <price_eur>65 EUR</price_eur> <!-- Use only if you want to index prices in several currencies -->
    <price_eur_old>67.50 EUR</price_eur_old> <!-- Use only if you want to index prices in several currencies -->
    <image_link_s>https://example.org/images/thumbs/100x100/2172.png</image_link_s>
    <image_link_m>https://example.org/images/thumbs/200x200/2172.png</image_link_m>
    <image_link_l>https://example.org/images/thumbs/600x600/2172.png</image_link_l>
    <description><![CDATA[A nice & comfortable shirt. It's ok to use <strong>html tags</strong> in description, as long as you wrap the contents inside CDATA directive.]]></description>
    <labels><![CDATA[Summer sale, Free shipping]]></labels>
    <product_code>NK8277S</product_code>
    <ean>8288881881818</ean>
    <to_cart_id>2172</to_cart_id>
    <margin>0.42</margin>
    <boost>1</boost> <!-- You can control boosting via the boost tag -->
    <introduced_at>2021-07-15</introduced_at>
    <parameters>
      <param>
        <name><![CDATA[Size]]></name> <!-- avoid using dots in names, e.g. "N. Size" is not a valid name -->
        <value><![CDATA[XS, M, L, XL]]></value>
      </param>
      <param>
        <name><![CDATA[Material]]></name>
        <value><![CDATA[Cotton (80%), Polyester (20%)]]></value>
      </param>
    </parameters>
  </item>
  <!-- more products/items -->
</items>

Luigi's Box accepts data from an arbitrary data feed structure, though to make the integration process smooth and fast, we recommend that you prepare the data in the Luigi's Box feed structure. Every ecommerce store is different, has different data and requirements, so keep the above information in mind when deciding what to put into the feed.

The Luigi's Box product feed format contains a set of required "tags" (attributes), which should be applicable regardless of the kind of products that you sell. For each product, you should also supply parameters in the flexible name/value format. You can use arbitrary names with a single rule - avoid dots in the parameter name.

Variants in the feed

Lugi's Box Search has a mode called "variants search", where we aggregate individual variants, and only show best match for each variant group. To activate this feature, we require that the individual variants are linked in the feed using item_group_id attribute and that they are listed in the feed together, i.e., they go one after another without being separated by any other unrelated product.

Example of XML feed if you are using product variants

<?xml version="1.0" encoding="UTF-8"?>
<items>
  <!-- Each item contains extra item_group_id attribute which links variants -->
  <item>
    <identity>NH7636A</identity>
    <title>Black Nike Shirt</title>
    <url>https://example.org/2172-black-nike-shirt</url>
    <item_group_id>8272</item_group_id>
    <!-- other attributes, excluded for brevity -->
  </item>
  <item>
    <identity>NH7636B</identity>
    <title>White Nike Shirt</title>
    <url>https://example.org/2173-white-nike-shirt</url>
    <item_group_id>8272</item_group_id>
  </item>
  <item>
    <identity>NH7636C</identity>
    <title>Red Nike Shirt</title>
    <url>https://example.org/2ii174-red-nike-shirt</url>
    <item_group_id>8272</item_group_id>
  </item>
  <!-- All variants of the same product MUST follow each other, variant by variant in the feed -->
  <item>
    <identity>NM2887A</identity>
    <title>Black Hoodie</title>
    <url>https://example.org/2175-black-hoodie</url>
    <item_group_id>8273</item_group_id>
  </item>
  <!-- more products/items -->
</items>

Your existing feeds

You may already have some data feeds ready that you use elsewhere and they may be useable for search. Below is a table of feeds that we frequently encounter and their usability.

Category feeds

This feed lives on your site, e.g. at https://example.org/feeds/categories.xml

<?xml version="1.0" encoding="UTF-8"?>
<categories>
  <category>
    <identity>1</identity>
    <name>Shirts</name>
    <url>https://example.org/categories/shirts</url>
  </category>
  <category>
    <identity>2</identity>
    <name>Blazers</name>
    <url>https://example.org/categories/blazers-men</url>
    <hierarchy>Apparel | Men</hierarchy>
  </category>
  <category>
    <identity>3</identity>
    <name>Blazers</name>
    <url>https://example.org/categories/blazers-women</url>
    <hierarchy>Apparel | Women</hierarchy>
  </category>
  <!-- more categories -->
</categories>

Category feeds are simpler than product feeds and we only require 2 attributes: title and URL.

The feed must be flat, no nesting is allowed (i.e., <category> nested in another <category> tag). If your categories are nested in a category tree, use a separate hierarchy tag to denote parent categories. Make sure that the name + hierarchy matches the categories that you use in the product feed to ensure correct pairing.

We can also process feeds in other formats (even your custom format), contact us to discuss.

Brand feeds

This feed lives on your site, e.g. at https://example.org/feeds/brands.xml

<?xml version="1.0" encoding="UTF-8"?>
<brands>
  <brand>
    <identity>7</identity>
    <name>NiceShirts</name>
    <url>https://example.org/brands/nice-shirts</url>
  </brand>
  <brand>
    <identity>8</identity>
    <name>Blue</name>
    <url>https://example.org/brands/blue</url>
  </brand>
</brands>

Brand feeds are very similar to categories. Make sure that the brand name matches the brand that you use in the product feed to ensure correct pairing.

Articles feeds

This feed lives on your site, e.g. at https://example.org/feeds/articles.xml

<?xml version="1.0" encoding="UTF-8"?>
<articles>
  <article>
    <identity>37271</identity>
    <name>Lorem ipsum title of the blog post</name>
    <annotation>Short description, perex</annotation>
    <text>Text of the article</text>
    <url>https://example.org/article/blog-post-lorem</url>
  </article>
  <article>
    <identity>37272</identity>
    <name>Lorem ipsum title of the article</name>
    <annotation>Short description, perex</annotation>
    <text>Text of the article</text>
    <url>https://example.org/brands/blog-post-ipsum</url>
  </article>
</articles>

Articles feeds are very similar to categories and brands.

We can also process feeds in other formats (even your custom format), contact us to discuss.

Add to cart

If you are planning to use managed Luigi's Box Search interface and want to enable "Add to cart" functionality directly from the search results page, we need an "interface" on your side to manage the actual adding to cart and any UI interaction it involves.

Please refer to the solution below for more detail.

Dynamic pricing

If your store uses dynamic pricing, i.e. different users may see different pricing for the same product, refer to the solutions below. Depending on the complexity of your scenario, you may need to provide all of the pricing data, or implement a pricing API.

Multiple product feeds

Luigi's Box is able to join and merge product data from several feeds to build the full product data. There are many situations when exporting a single product feed with all of the data may be problematic, either because of performance problems associated with taking a full data snapshot, or with limitations when accessing multiple data stores on your side.

To use the feature, you need to prepare a single driving feed and then up to 4 additional feeds. Lugi's Box will then process the driving feed and for each product, lookup and merge product data from the additional feeds.

Usually the driving feed will contain product data that is not changing frequently, such as product descriptions, link to images or product parameters. To save performance on your side, you may only regenerate this feeds once per day. The dynamic data, such as pricing or availability will be present in the second feed, which you will regenerate hourly.

Luigi's Box can combine up to 5 different feeds to build full product data. The feature depends on a single "join attribute" being present in the driving feed and in all of the joined feeds. This "join attribute" can be an arbitrary attribute that will be used to uniquely identify and match product data across all feeds. Most frequently, you will use product ID.

Updating data via feeds and API simultaniously

While it is technically possible to provide product updates using feeds and API at the same time, it is an approach that we do not recommend as it will lead to subtle and hard-to-debug data discrepancies. The intuitive expectation is that you may be able to provide Luigi's Box a feed with the product data that will be processed periodically and then you will use the Partial update API to update frequently changing product attributes such as pricing or availability in real time.

This approach will however, lead to unexpected data inconsistencies. The feed processing is not instant, but rather takes some time depending on the number of products, structure of the data and other factors. The feed processing mechanism downloads the feed — a snapshot of your data at the time of the download — and processes it product by product. It is possible that you update the product using API and then minutes later, the feed processing mechanism which is in progress will overwrite the product data back to the state that it was found in the feed at the time of the download. Luigi's Box currently doesn't provide any protection against this kind of data inconsistency so the combination of feeds and API is best avoided.

As an alternative to this combination, you may consider simply increasing feeds processing frequency, possibly combined with splitting product data into several feeds to decrease the load on your side.