Overview Of Impex
Impex stands for Import and export
As the name suggests, Impex in hybris is used for importing data from CSV file / impex file to hybris system and exporting data from hybris system to CSV file.
Hybris has come up with an extension called “Impex” inside platform/ext folder which helps in Inserting,updating,deleting and exporting data.
Impex import means always data flows into Hybris system.
Impex export means always data flows out of Hybris system
Why do we need to import ?
We need to import data multiple times into Hybris system.
As we all know that any ecommerce site will have a data like products,customer,regions,countries,promotions etc…
All this data has to be inserted into Hybris system to display it on the site.
So to insert this kind of data into hybris system, we go for Impex Import.
We can insert data either at the run time or during initialization or during update process.
Why do we need to export?
Whenever we need to take the backup of data, we have to export the data into csv files so that later it can be imported into Hybris system.
If we are doing initialization,it is recommended to take data backup using Impex Export.
Once the initialization is done, we can import the data back to hybris system.
Impex jargons
1) Header
2) Value lines
3) Macros
4) Comments
Header
Header in impex will have below structure
- Mode itemType[modifier=value];attribute1[modifier=value];attribute2[modifier=value];attribute3[modifier=value]
Mode itemType[modifier=value];attribute1[modifier=value];attribute2[modifier=value];attribute3[modifier=value]
We can see below things in header
Mode
It specifies what operation we are going to perform on the values(Insert,update,remove)
There are 4 mode types possible in Header
1) INSERT
2) UPDATE
3) INSERT_UPDATE
4) REMOVE
1) INSERT :
This Mode is used to insert a new record into Database.
If the record already exists it will throw an exception.
2) UPDATE :
This Mode is used to update the existing record.
If the record does not exist then exception will be thrown.
3) INSERT_UPDATE :
This Mode is the most preferred one for insertion or update.
This header will insert the record if its new one and update the record if it is already exists.
So there will be no exception if we use this mode.
4) REMOVE :
This mode is used to remove the existing records.
We need to specify one unique attribute to remove the record.
Type
specifies the item type on which the Mode is applied (Example:Product,Category,Customer etc )
Basically it impacts the table associated with this Type.
Attributes
It defines the attributes of the item type which are defined in items.xml file.
Each attribute specified will get the values affected on corresponding columns in DB.
Modifier
It is used to specify the additional information for the attributes like specifying attribute uniqueness using unique=true.
We can specify multiple modifers in 2 ways
1) Comma separated modifiers within single bracket [] as below
- attribute[modifier1=value1,modifier2=value2,modifier3=value3];
attribute[modifier1=value1,modifier2=value2,modifier3=value3];
Single bracket has all the modifiers separated by comma.
2) each modifier with separate bracket [] as below
- ;attribute[modifier1=value1][modifier2=value2][modifier3=value3];
;attribute[modifier1=value1][modifier2=value2][modifier3=value3];
Separate bracket for each modifier
Example:
- INSERT_UPDATE category;code[unique=true];name[lang=de];name[lang=en];$supercategories;$thumbnail;description[lang=de];order
INSERT_UPDATE category;code[unique=true];name[lang=de];name[lang=en];$supercategories;$thumbnail;description[lang=de];order
In the above header line,
INSERT_UPDATE is the mode
Category is the item type
Code,name etc are the attributes
[unique=true] is the modifier
Note:
This header is applicable for all the value lines till the next header appears in the file.
One file can have any number of headers.
Value lines in Impex
There could be one or more value lines in the file.
Each value line represents one row of data in table.
Each field in the value line is separated by semicolon and represents the data into one column.
Example:
Field1;field2;field3
Space is also considered as value of the field.
Each line should have the same number of fields.
If field contains any line breaks or quotes, then enclose such fields in double quotes.
Macros
Macros are basically used to define the value at one place and use the macro in all the places.
It helps to eliminate writing same lines again and again.
Macros starts with “$” and they are referenced by $macroName
Example:
- $catalog=catalog(id)
$catalog=catalog(id)
Defined a macro called “catalog” and can be referenced as $catalog wherever required.
- $catalogVersion=catalogVersion($catalog,version)
$catalogVersion=catalogVersion($catalog,version)
Defined a macro called “catalogVersion” and can be referenced as $catalogVersion wherever required.
- INSERT Product; code; $catalogVersion
INSERT Product; code; $catalogVersion
Now the macros $catalogVersion and $catalog are replaced by their actual value as below
- catalogVersion(catalog(id),version)
catalogVersion(catalog(id),version)
If we don’t use macros then we have to write above line in all the places wherever required.
Comments
Comments in impex file starts with # symbol.
Example:
- #Inserting product details
- INSERT_UPDATE Product;code[unique=true]; varianttype(code);name[lang=en];
#Inserting product details INSERT_UPDATE Product;code[unique=true]; varianttype(code);name[lang=en];
In the above lines, we have one comment line starts with # which will be ignored while importing.
Is there a way to export data like Product/Customer data from Hybris DB which contains more than 500000 records?
Hi Kb,
How to remove data from enumtype?
what is Translator and processor?
Hi,
Its very useful article for beginners and let us know all about IMPEX can you guys add more about CSV also.
Regards,
Mangesh
Thank you for the article.
I noticed one small inconsistency – in value lines, the values are separated by semicolons by default, NOT commas.
Everything else looks good 🙂
Thanks again,
George
Updated , Thanks George !!
How can i include # in my attribute values
Hi KB,
Thank you for all the efforts you are putting into to teach people Hybris in most simple way. 🙂
I have a question regarding IMPEX, how do we choose between Translator, Decorator and Processor?
Thanks In Advance,
Rupesh
I am importing impex through code. Any idea on passing collectiontype in attribute?
Wow…good explanation…..
Why we use Translators in impex, and how we can modify impex at run time.
Very good and detail explanation KB. Thank you so much.
Thanks and Happy learning 🙂
can you give more detail about impex
like real time example for export and import
outstanding job KB, thanks for sharing your knowledge.
I just got to know this site, would be going through the same.
Thanks Saurabh , Happy learning 🙂
Hi KB,
Your document is sufficient for beginner Hybris but can you please upload docs related with OMS, CATALOG, PROMOTION, VOUCHERS and
i have one doubt: what is the difference between translator & shell script in impex and how to use?
Thanks in advance:)
How to make a most popular product by using Impex? What is the syntax for that?
It depends on your approach or design , You can define one flag or counter at the product level and populate it using product impex and write business code accordingly.
Can we have more than 2 versions(staged and online)? If yes, please explian how.
Thanks in advance!
You can have as many versions as you want, but its recommended to have Staged and Online versions.
how to export data?
You just need to write the export script specifying the type to export and run it in HAC
How to write export script? Can you please explain? I am not getting it.
I will write article on this separately, check in wiki for time being
Hi KB Sir,
we have two versions of catalog like staged and online,why we require these two versions…?what happens if i change something like price directly in online version..?
Hi Sravan,
Staged version is required to verify the data before sending it to online version
Because what we see in online is coming from online version.
Modified content directly in online version will be displayed on the site with no issues but its not a good practice as control of verifying the content is lost here.
Thanks you so much KB for your valuable information
You are welcome Sravan !!
Hi KB,
I have a basic question, for retrieving data we have flexible search or solr and for data load we have IMPEX. How can can we insert data at run time? like a user is registration
Hello,
For inserting data at run time ,You can use Model class persistent mechanism where you will populate all the data from UI to your model object and then call save method of ModelService to persist the same in DB.
As per my understanding, macros are like variables with certain values that can be referenced anywhere in impex using its name with $ symbol.
My question is why macros are refernced in HEADER section ?
Ideally header section must have ItemType and list of attributes separated by semicolon (;)
macros are not any attribute they are values then they should be in values section.
Can you please clear my doubt ?
Anish : Your understanding is right.
As stated in this article, Macros are basically used to define the value at one place and use them in all the places.
Its just a replacement of value in the place where we are using it.
Ex: if my macro is declared as $x=1
then x will be replaced by 1 in the place where we reference it.
So if we are referencing the macros at the header then macro must be defined with attribute name as well so that Header will still follow your understanding
ItemType and list of attributes separated by semicolon (;)
If Macro is defined with just value without attribute, then such Macro can not be referenced in the Header.
Hope you understand it.
Let me know if you still have any doubt.
Thanks!!
Can you tell me the what is the use &componentRef and $contentCV in impex files..?
“&” is used to refer the part of attribute in impex which means that referred attribute is a part of item type which is referring it.
“$contentCV” – its a macro used to define catalog(like content or product) and version(like staged or online) as a composite unique key.
Thank you so much KB…:)
You are welcome !!
Can you tell me why do we need to provide the modifier unique=true in the impex file for updating records, even if that attribute is already declared as unqiue=true in the items.xml file ?
In impex, it is not necessary that only fields which are declared as unique = true in items.xml file has to be unique.
Any field can be declared as unique in impex and accordingly data will be updated.
Ex: In Product if code is defined as unique = true in items.xml but we can write an impex to make product name as unique and update the record through product name even though it is not defined as unique in items.xml.
So Impex file behaves differently with Unique modifier than what we specify in items.xml, hence we need to tell Impex API that which one it has to pick up to find uniqueness and update.
awesome Kb….
Thank you Himanshu !!
can you please explain me what exactly is code and item and difference between them
Code is the unique identifier to represent the type, type is any item we define to represent some attributes(ex:enum type,item type etc)
hi kb,
can u please tell me how to remove particular attribute through impex.