Understanding Solr configuration in Hybris
In Hybris, we have to configure the Solr in impex files
Lets see the impex files used for the same in detail
1)solr.impex
/import/coredata/stores/hybris/solr.impex
2)solr_en.impex
/import/coredata/stores/hybris/solr_en.impex
3)solrtrigger.impex
import/coredata/stores/hybris/solrtrigger.impex
1)solr.impex
This file contains all the configuration of Solr including Solr server,indexing types,facets etc.
Note: All the variables in the below impex files which starts with $ are called as macros and should be defined at the beginning of the impex.
Solr server configuration
We need to insert the data to SolrServerConfig table with solr server configuration name and mode of the server to indicate whether it’s embedded or standalone server.
Below impex line will do this
- INSERT_UPDATE SolrServerConfig;name[unique=true];mode(code);embeddedMaster
- ;$serverConfigName;embedded;true
INSERT_UPDATE SolrServerConfig;name[unique=true];mode(code);embeddedMaster ;$serverConfigName;embedded;true
We have configured solr server to be embedded.
If we want to make solr server as standalone then we have to insert below data to SolrServerConfig table
- INSERT_UPDATE SolrServerConfig;name[unique=true];mode(code);embeddedMaster
- ;$serverConfigName;standalone;false;
INSERT_UPDATE SolrServerConfig;name[unique=true];mode(code);embeddedMaster ;$serverConfigName;standalone;false;
For standalone solr server configuration , we must insert data to SolrEndpointUrl table as well to indicate the actual url where solr is running.
- INSERT_UPDATE SolrEndpointUrl;solrServerConfig(name)[unique=true];url[unique=true];master[unique=true,default=false]
- ;$serverConfigName;http://localhost:8983/solr;true
INSERT_UPDATE SolrEndpointUrl;solrServerConfig(name)[unique=true];url[unique=true];master[unique=true,default=false] ;$serverConfigName;http://localhost:8983/solr;true
solr indexer configuration
we need to insert the data to SolrIndexConfig table with index configuration name,batch size of indexing,number of threads to run and the index mode
- INSERT_UPDATE SolrIndexConfig;name[unique=true];batchSize;numberOfThreads;indexMode(code);
- ;$indexConfigName;100;1;TWO_PHASE;
INSERT_UPDATE SolrIndexConfig;name[unique=true];batchSize;numberOfThreads;indexMode(code); ;$indexConfigName;100;1;TWO_PHASE;
We have defined the batch size to be 100
1 thread for processing index and
TWO_PHASE as the indexing strategy.
Check Solr overview in Hybris article for the details of TWO_PHASE indexing mode.
Solr search configuration
we need to insert the data to SolrSearchConfig table with page size and description
- INSERT_UPDATE SolrSearchConfig;description[unique=true];pageSize
- ;$searchConfigName;20
INSERT_UPDATE SolrSearchConfig;description[unique=true];pageSize ;$searchConfigName;20
We have given the page size as 20, so 20 pages of results will be retrieved by Solr.
Define the Solr indexed Types
We need to define the hybris item types to be indexed in the SolrIndexedType table
- INSERT_UPDATE SolrIndexedType;identifier[unique=true];type(code);variant;sorts(&sortRefID)
- ;$solrIndexedType;Product;false;sortRef1,sortRef2,sortRef3,sortRef4,sortRef5,sortRef6
INSERT_UPDATE SolrIndexedType;identifier[unique=true];type(code);variant;sorts(&sortRefID) ;$solrIndexedType;Product;false;sortRef1,sortRef2,sortRef3,sortRef4,sortRef5,sortRef6
We have defined only one hybris item type to be indexed which is Product Type and defined some of the sorting references which we will define in the subsequent impex below.
Solr Facet Search Config
We need to insert the data to SolrFacetSearchConfig table to define index name prefix,supporting languages,currencies and also link the solrServerConfig, solrSearchConfig, solrIndexConfig, solrIndexedTypes defined above.
We also need to define the fallback language and catalog version
- INSERT_UPDATE SolrFacetSearchConfig;name[unique=true];description;indexNamePrefix;languages(isocode);currencies(isocode);solrServerConfig(name);solrSearchConfig(description);solrIndexConfig(name);solrIndexedTypes(identifier);enabledLanguageFallbackMechanism;$catalogVersions
- ;$facetSearchConfigName;$facetSearchConfigDescription;$searchIndexNamePrefix;$indexLanguages;$indexCurrencies;$serverConfigName;$searchConfigName;$indexConfigName;$solrIndexedType;true;$productCatalog:Online,$productCatalog:Online
INSERT_UPDATE SolrFacetSearchConfig;name[unique=true];description;indexNamePrefix;languages(isocode);currencies(isocode);solrServerConfig(name);solrSearchConfig(description);solrIndexConfig(name);solrIndexedTypes(identifier);enabledLanguageFallbackMechanism;$catalogVersions ;$facetSearchConfigName;$facetSearchConfigDescription;$searchIndexNamePrefix;$indexLanguages;$indexCurrencies;$serverConfigName;$searchConfigName;$indexConfigName;$solrIndexedType;true;$productCatalog:Online,$productCatalog:Online
Link the solr to the site
We need to link the solr configuration to the site by providing solrFacetSearchConfiguration to the Base site as below
- UPDATE BaseSite;uid[unique=true];solrFacetSearchConfiguration(name)
- ;$indexBaseSite;$facetSearchConfigName
UPDATE BaseSite;uid[unique=true];solrFacetSearchConfiguration(name) ;$indexBaseSite;$facetSearchConfigName
Defining the range value set
We need to insert the data to SolrValueRangeSet table for defining the Solr value range references which we will define in the next impex.
- INSERT_UPDATE SolrValueRangeSet;name[unique=true];qualifier;type;solrValueRanges(&rangeValueRefID)
- ;priceRangeGBP;GBP;double;rangeRefGBP1,rangeRefGBP2,rangeRefGBP3,rangeRefGBP4,rangeRefGBP5
INSERT_UPDATE SolrValueRangeSet;name[unique=true];qualifier;type;solrValueRanges(&rangeValueRefID) ;priceRangeGBP;GBP;double;rangeRefGBP1,rangeRefGBP2,rangeRefGBP3,rangeRefGBP4,rangeRefGBP5
We have defined 5 ranges for priceRangeGBP and we will define the values for the same in the below impex
We need to insert the data for SolrValueRange table to define the actual range values for each range reference as below
- INSERT_UPDATE SolrValueRange;&rangeValueRefID;solrValueRangeSet(name)[unique=true];name[unique=true];from;to
- ;rangeRefGBP1 ;priceRangeGBP;£0-£19.99; 0; 19.99
- ;rangeRefGBP2 ;priceRangeGBP;£20-£49.99; 20; 49.99
- ;rangeRefGBP3 ;priceRangeGBP;£50-£99.99; 50; 99.99
- ;rangeRefGBP4 ;priceRangeGBP;£100-£199.99; 100;199.99
- ;rangeRefGBP5 ;priceRangeGBP;£200-£299.99; 200;299.99
INSERT_UPDATE SolrValueRange;&rangeValueRefID;solrValueRangeSet(name)[unique=true];name[unique=true];from;to ;rangeRefGBP1 ;priceRangeGBP;£0-£19.99; 0; 19.99 ;rangeRefGBP2 ;priceRangeGBP;£20-£49.99; 20; 49.99 ;rangeRefGBP3 ;priceRangeGBP;£50-£99.99; 50; 99.99 ;rangeRefGBP4 ;priceRangeGBP;£100-£199.99; 100;199.99 ;rangeRefGBP5 ;priceRangeGBP;£200-£299.99; 200;299.99
we will use priceRangeGBP to provide range for the indexed property while defining indexed property.
Defining the non-facet indexed properties
We need to insert the data into SolrIndexedProperty table for defining the properties of item types to be indexed.
We need to define different features for each property on how it should be indexed, whether it has to be multivalued,used for autocomplete etc.
- INSERT_UPDATE SolrIndexedProperty;solrIndexedType(identifier)[unique=true];name[unique=true];type(code);sortableType(code);currency[default=false];localized[default=false];multiValue[default=false];useForSpellchecking[default=false];useForAutocomplete[default=false];fieldValueProvider;valueProviderParameter
- ;$solrIndexedType; name ;text ;sortabletext; ;true; ;true;true;
- ;$solrIndexedType; priceValue ;double ; ;true; ; ; ; ;productPriceValueProvider;
INSERT_UPDATE SolrIndexedProperty;solrIndexedType(identifier)[unique=true];name[unique=true];type(code);sortableType(code);currency[default=false];localized[default=false];multiValue[default=false];useForSpellchecking[default=false];useForAutocomplete[default=false];fieldValueProvider;valueProviderParameter ;$solrIndexedType; name ;text ;sortabletext; ;true; ;true;true; ;$solrIndexedType; priceValue ;double ; ;true; ; ; ; ;productPriceValueProvider;
We also need to define the Value provider for some of the attributes whose value cannot be understood by Solr directly.
We will see how value providers works in a separate article.
Defining the Facet indexed properties
Facet properties are generally displayed at the sidebar of the site where we can select those facet fields to get the list of products accordingly.
- INSERT_UPDATE SolrIndexedProperty;solrIndexedType(identifier)[unique=true];name[unique=true];type(code);sortableType(code);currency[default=false];localized[default=false];multiValue[default=false];facet[default=true];facetType(code);facetSort(code);priority;visible;useForSpellchecking[default=false];useForAutocomplete[default=false];fieldValueProvider;facetDisplayNameProvider;customFacetSortProvider;topValuesProvider;rangeSets(name)
- ;$solrIndexedType; price ;double ; ;true ; ; ; ;MultiSelectOr ;Alpha ; 4000;true; ; ;productPriceValueProvider ; ; ;defaultTopValuesProvider ;priceRangeGBP
INSERT_UPDATE SolrIndexedProperty;solrIndexedType(identifier)[unique=true];name[unique=true];type(code);sortableType(code);currency[default=false];localized[default=false];multiValue[default=false];facet[default=true];facetType(code);facetSort(code);priority;visible;useForSpellchecking[default=false];useForAutocomplete[default=false];fieldValueProvider;facetDisplayNameProvider;customFacetSortProvider;topValuesProvider;rangeSets(name) ;$solrIndexedType; price ;double ; ;true ; ; ; ;MultiSelectOr ;Alpha ; 4000;true; ; ;productPriceValueProvider ; ; ;defaultTopValuesProvider ;priceRangeGBP
We have defined price field as facet field and it can be selected to filter the products based on that field.
we have defined facet[default=true] to indcate that the field is facet type
Define the indexed queries
We need to define the indexed queries for full index and update index so that solr picks up the corresponding source data from hybris for indexing.
- INSERT_UPDATE SolrIndexerQuery;solrIndexedType(identifier)[unique=true];identifier[unique=true];type(code);injectCurrentDate[default=true];injectCurrentTime[default=true];injectLastIndexTime[default=true];query;user(uid)
- ;$solrIndexedType;$solrIndexedType-fullQuery;full;;;false;"SELECT {PK} FROM {Product}";anonymous
- ;$solrIndexedType;$solrIndexedType-updateQuery;update;;;;"SELECT {p:PK} FROM {Product AS p} WHERE ({p:modifiedtime} >= ?lastIndexTime";anonymous
INSERT_UPDATE SolrIndexerQuery;solrIndexedType(identifier)[unique=true];identifier[unique=true];type(code);injectCurrentDate[default=true];injectCurrentTime[default=true];injectLastIndexTime[default=true];query;user(uid) ;$solrIndexedType;$solrIndexedType-fullQuery;full;;;false;"SELECT {PK} FROM {Product}";anonymous ;$solrIndexedType;$solrIndexedType-updateQuery;update;;;;"SELECT {p:PK} FROM {Product AS p} WHERE ({p:modifiedtime} >= ?lastIndexTime";anonymous
We have defined the 2 queries that Solr uses for full indexing and update indexing.
Full indexing query gets all the data without any condition
Update indexing query gets the data which has been modified from the last indexed time, so it won’t fetch entire data for indexing and hence it’s faster too.
Defining the Sort
We need to insert the sort references into SolrSort table which we will be using while inserting values for SolrIndexedType to define the sorts for the indexed item type
- INSERT_UPDATE SolrSort;&sortRefID;indexedType(identifier)[unique=true];code[unique=true];useBoost
- ;sortRef1;$solrIndexedType;relevance;true
- ;sortRef2;$solrIndexedType;topRated;false
- ;sortRef3;$solrIndexedType;name-asc;false
- ;sortRef4;$solrIndexedType;name-desc;false
- ;sortRef5;$solrIndexedType;price-asc;false
- ;sortRef6;$solrIndexedType;price-desc;false
INSERT_UPDATE SolrSort;&sortRefID;indexedType(identifier)[unique=true];code[unique=true];useBoost ;sortRef1;$solrIndexedType;relevance;true ;sortRef2;$solrIndexedType;topRated;false ;sortRef3;$solrIndexedType;name-asc;false ;sortRef4;$solrIndexedType;name-desc;false ;sortRef5;$solrIndexedType;price-asc;false ;sortRef6;$solrIndexedType;price-desc;false
we have given these references while inserting data into SolrIndexedType table above.
Define the Solr sort fields
We need to insert the sorting fields into SolrSortField table which will be linked to the sort reference codes defined above
- INSERT_UPDATE SolrSortField;sort(indexedType(identifier),code)[unique=true];fieldName[unique=true];ascending[unique=true]
- ;$solrIndexedType:relevance;inStockFlag;false
- ;$solrIndexedType:relevance;score;false
- ;$solrIndexedType:topRated;inStockFlag;false
- ;$solrIndexedType:topRated;reviewAvgRating;false
- ;$solrIndexedType:name-asc;name;true
- ;$solrIndexedType:name-desc;name;false
- ;$solrIndexedType:price-asc;priceValue;true
- ;$solrIndexedType:price-desc;priceValue;false
INSERT_UPDATE SolrSortField;sort(indexedType(identifier),code)[unique=true];fieldName[unique=true];ascending[unique=true] ;$solrIndexedType:relevance;inStockFlag;false ;$solrIndexedType:relevance;score;false ;$solrIndexedType:topRated;inStockFlag;false ;$solrIndexedType:topRated;reviewAvgRating;false ;$solrIndexedType:name-asc;name;true ;$solrIndexedType:name-desc;name;false ;$solrIndexedType:price-asc;priceValue;true ;$solrIndexedType:price-desc;priceValue;false
We have defined 5 sort fields as below
inStockFlag field can be sorted based on relevance and Top rating in descending order.
score field can be sorted based on relevance only in descending order
reviewAvgRating field can be sorted based on Top rating only in descending order
name field can be sorted based on both ascending & descending order
price field can be sorted based on both ascending & descending order
2)solr_en.impex
import/coredata/stores/apparel-uk/solr_en.impex
This impex file is used for localization of solr indexed fields in English language as below
- UPDATE SolrIndexedProperty;solrIndexedType(identifier)[unique=true];name[unique=true];displayName[lang=$lang]
- ;$solrIndexedType;allPromotions;"Promotions"
- UPDATE SolrSort;indexedType(identifier)[unique=true];code[unique=true];name[lang=$lang]
- ;$solrIndexedType;name-asc;"Name (ascending)"
UPDATE SolrIndexedProperty;solrIndexedType(identifier)[unique=true];name[unique=true];displayName[lang=$lang] ;$solrIndexedType;allPromotions;"Promotions" UPDATE SolrSort;indexedType(identifier)[unique=true];code[unique=true];name[lang=$lang] ;$solrIndexedType;name-asc;"Name (ascending)"
Similarly for other languages we will have corresponding solr localization files in the appropriate localization folders.
3)solrtrigger.impex
/import/coredata/stores/apparel-uk/solrtrigger.impex
This impex file is used to schedule the cron jobs for Solr indexing.
- INSERT_UPDATE Trigger;
- cronJob(code)[unique=true];second;minute;hour;day;month;year;relative;active;maxAcceptableDelay
- # Run the full index cronJob at 3:05 AM every day
- ;full-index-cronJob;0;5;3;-1;-1;-1;false;false;-1
- # Run the update index cronJob every 1 minutes
- ;update-index-cronJob;0;1;-1;-1;-1;-1;true;false;-1
INSERT_UPDATE Trigger; cronJob(code)[unique=true];second;minute;hour;day;month;year;relative;active;maxAcceptableDelay # Run the full index cronJob at 3:05 AM every day ;full-index-cronJob;0;5;3;-1;-1;-1;false;false;-1 # Run the update index cronJob every 1 minutes ;update-index-cronJob;0;1;-1;-1;-1;-1;true;false;-1
We have defined cron job scheduling time through Trigger table for both full and update indexing.
We can observe that full indexing job is scheduled only once a day and update indexing is scheduled to run every 1 minute.
we already know the reason why update index runs more frequently, if not please check Solr overview in Hybris article
For practicing in B2C Apparel store, check the below impex file
hybris\bin\ext-data\apparelstore\resources\apparelstore\import\coredata\stores\apparel-uk\solr.impex
isn’t that cron-expression runs after every 1hr in update-index-cronjob, i think it’s not after every 1 minute.
Hi KB,
Please help me with this issue!!!
after following all the steps, when I am trying to search products using alias keyword, I am getting alias for blue cap in auto suggestions but when I click on it, I am not getting any product results(PLP page) wheraeas I am able to see in solr. Please assist.
Thanks
Nice Article and very informative.
Good work KB
Hi KB,
Thanks for sharing this information.how to display facets according to usergroups ? waiting for reply.
Best Regards,
Aniket Deshmukh.
i am indexing new itemType which doesnt extends from Product.
i defined new item type in XXXX-items.xml and then created new Index type and associate it with existing SOLR Facet configuration. indexing ran fine but i cant see my results in SOLR result.
i created new index type by following steps in hybris help:
https://help.hybris.com/6.7.0/hcd/bc912462e6974ac596b252cd18680e4e.html#loio875217e3c7f44580a5ddb501862e8760
can please you advise what i could be missing?
i can see the documents in solr. its a new core thats why i was not seeing those earlier. but now any idea how to use new core for displaying search result along with existing core. means i need to search result from product(OOB) core and my new core created for my item type.
Hi,
We have upgraded from hybris version 5.6 to 6.7.
In case for solr , I have followed the above steps, but the indexer pop up doesnt open in the backoffice.
There is no error on console.
Also, ton running the trigger impex here , i was getting an error that the cronjob is not available.Thus, i have created a cronjob and assigned the trigger to it.
Do you have any idea what i might be missing and why the indexer pop doesnt open?
Hi KB,
I wanted to show different facets based on usergroup, do you have any idea how to achieve this??
Thanks,
Siva
Have you find any solution for your query.
Could you help me ?
In the below solr.impex why we use “text” type(code) for SolrIndexedProperty “name”.
INSERT_UPDATE SolrIndexedProperty;solrIndexedType(identifier)[unique=true];name[unique=true];type(code);sortableType(code);currency[default=false];localized[default=false];multiValue[default=false];useForSpellchecking[default=false];useForAutocomplete[default=false];fieldValueProvider;valueProviderParameter
;$solrIndexedType; name ;text ;sortabletext; ;true; ;true;true;
Good work Karibasappa. Greatly appreciate. This will help a lot.
Thank you Siddesh !!
Hi KB,
Anytime I search for something related to Hybris, your page comes up! I can find better explanations than on Hybris Wiki. Great job!!
Now, I am curious about something concerning Solr in Hybris. Let’s say I have a facet in solr that is a classification attribute. It’s values are true and false. Is is possible to have the products with a value false retrieved at the end? I tried low priority but it didn’t take. Any ideas?
Thanks!
Hi,Thank you !!
If you make facet as false, then product will be retrieved but attribute will not be available as facet value.
Let me know if there is something different you are looking for.
Hi KB,
here is the following impex
INSERT_UPDATE SolrSort;&sortRefID;indexedType(identifier)[unique=true];code[unique=true];useBoost
;sortRef1;$solrIndexedType;relevance;true
;sortRef3;$solrIndexedType;name-asc;true
;sortRef4;$solrIndexedType;name-desc;true
;sortRef5;$solrIndexedType;price-asc;true
;sortRef6;$solrIndexedType;price-desc;true
when I set useBoost=true for name-asc, name-desc, price-asc, price-desc and relevance – the products are sorted in the same order for all of these types of sorting
I have no boost rules defined for products… so why name and price are not sorted when useBoost=true ?
Hi KB,
In which scenario we will go through embedded mode and for which scenario we will go through standalone mode.Could you please clarified it.
Nice article!
However, I have encountered an issue with items getting indexed. On my end, it gets all the items regardless of the CatalogVersion even if we specify the it in the Facet Search Config. Do you have any idea on this?
Thank you!
Hi Karibasappa,
I have some confusion in Solr and i’m new to Solr can you provide the you contact number.
Thanks
Rajesh
Why do we need to define sort. Can you explain that in realtime where we need it by example.
We need to define the sort fields if we are providing the Sort functionality to the user on some fields while searching for a product.
Example:
You can sort products based on its Popularity or based on its price
If you open jabong or flipkart site, You can see this Sort By option to sort the products.
Thank you for help,
there is paramater in SolrSort useBoost – can you explain why we need it.
INSERT_UPDATE SolrIndexConfig;name[unique=true];batchSize;numberOfThreads;indexMode(code);legacyMode
;$indexConfigName;100;1;TWO_PHASE;false
Hi, what is the good config for – numberOfThreads in your experience
Boost will help Solr to consider it more important while searching.
Its like Google Page Rank on the documents search.
There are 2 kinds of boosts. Index-time and Query-time boosts.
Index-time boosts are applied when adding documents, and apply to the entire document or to specific fields.
Query-time boosts are applied when constructing a search query, and apply to specific fields.
Thanks KB.
Hi KB,
To add to my previous question, another question related impex syntax
INSERT_UPDATE SolrIndexedType;identifier[unique=true];type(code);variant;sorts(&sortRefID)
;$solrIndexedType;Product;false;sortRef1,sortRef2,sortRef3,sortRef4,sortRef5,sortRef6
what does the ‘&’ symbol and its significance here
Thanks
Sai
Hi Sai,
“&” is used to define “PartOf” attributes in impex, its also called Document Id , which is used to reference the attributes defined in the same impex only.
Hi KB,
Your explanation is very interesting.
I have a query here that what does batchsize in solr indexer and it differ from pagesize??
Generally what would be the prefered number of threads configuration for solr indexer in different environments ??
Hi Sai,
Thank you!!
Batchsize specifies how many records solr can process at a time when it gets from actual DB for indexing…
If records are huge then batch size helps Solr to process it batch by batch.
Page size specifies how many pages of data can be retrieved by Solr.
Example:
If batch size is 100 and page size is 20 it means each page it can retrieve 100 records..
simply awesome explanation
Thank you!!
Great article on Solr and hybris integration!
Thank you Prem !!
Great article really helped me a lot . Keep it up.
Thank you Somdip !!