Supplier Feed Automation: Import Product Data Into ERP and Shop
Stop importing supplier CSV, XML and Excel files by hand. How to fetch, validate, map and dedupe supplier data, then sync prices and stock automatically.
- Written by
- Md AshikSenior Software Engineer, Artisan Dojo
- Published
- Reading time
- 8 min read
Quick summary30-second read
- Supplier feed automation is a scheduled pipeline that fetches, validates, normalizes and maps supplier files, then pushes only changes into your ERP, shop and marketplaces.
- Validate every feed before importing, and stop with an alert when row counts drop sharply or prices change unusually.
- Match products across suppliers by GTIN, normalized to 14 digits with check digits verified, never by title.
- Hash normalized records and publish only deltas, running stock and prices separately from product content, plus a scheduled full reconciliation.
- Treat supplier stock as out of date: apply buffers per supplier or category, and cap large quantities to avoid overselling.
Supplier feed automation means a scheduled pipeline fetches each supplier's product, price and stock files, validates and normalizes them, maps them to your own catalog structure, and pushes only what changed into your ERP, PIM, shop and marketplaces. Nobody opens a CSV in Excel anymore. Done well, it also catches bad supplier data before it reaches your customers.
This guide explains how such a pipeline is built and where it usually goes wrong.
Why is importing supplier feeds manually a problem?
Manual imports work with one supplier and a few hundred products. They stop working when catalogs grow:
- Stock goes stale. If stock is imported once a day by hand, you sell items the supplier ran out of hours ago.
- Prices drift. A supplier raises purchase prices and your selling prices stay the same until someone notices the margin is gone.
- Errors slip through. A shifted column, a wrong decimal separator or a missing EAN gets imported as is.
- The work does not scale. Every new supplier adds another file format, another column layout and another person-hour per week.
What formats do suppliers send product data in?
Expect a mix. Even suppliers in the same industry rarely agree on a format.
| Format | What it looks like in practice |
|---|---|
| CSV / TXT | The most common. Watch for different delimiters, encodings and decimal separators. |
| Excel (XLSX) | Common with smaller suppliers. Often contains merged cells, notes and changing column orders. |
| XML | Usually a supplier-specific schema; sometimes a standard such as BMEcat. |
| JSON / REST API | Larger suppliers and wholesalers. Best for frequent stock and price updates. |
| EDI | Mostly in established retail and wholesale relationships, for example EDIFACT PRICAT price catalogs. |
What is BMEcat?
BMEcat is an XML standard for exchanging electronic product catalogs, created and published by BME, the German association for materials management, purchasing and logistics. It is free to use and common in German-speaking B2B trade. A BMEcat file can carry product data, prices and classification values from systems such as ECLASS or ETIM. The standard defines separate transactions for a full catalog (T_NEW_CATALOG), product updates (T_UPDATE_PRODUCTS) and price updates (T_UPDATE_PRICES), although many suppliers simply send the full catalog every time.
How are supplier feeds delivered?
- HTTP(S) download: a fixed URL, sometimes protected with a token or basic auth.
- FTP or SFTP: the supplier drops files into a folder, or you pull from theirs. Prefer SFTP; plain FTP sends credentials unencrypted.
- Email attachments: still common with small suppliers. Use a dedicated mailbox and parse attachments automatically.
- API: you call the supplier's endpoints on a schedule, or they push updates to you.
Ask each supplier how often the file is regenerated; fetching more often than that is pointless.
How does a supplier feed pipeline work, step by step?
A reliable pipeline separates each concern into its own step, so a problem in one supplier's file does not break everything else.
- Fetch. Download the file or call the API on a schedule that matches how often the supplier updates it. Store the raw file with a timestamp, so you can always see exactly what the supplier sent.
- Validate. Check the structure before using the content: expected columns present, row count within a normal range, required fields filled, numbers parse as numbers. If a feed suddenly has 40% fewer rows than yesterday, stop and alert instead of importing it.
- Normalize. Convert every supplier's format into one internal model: same encoding (UTF-8), same units, same decimal format, same currency handling, trimmed text, consistent field names.
- Map categories and attributes. Translate supplier categories and attribute names ("Farbe", "colour", "col.") into your own taxonomy and attribute set. Keep these mappings in tables that someone can edit, not hardcoded in scripts.
- Enrich. Add what the supplier does not provide: your own titles and descriptions, images from a media server, marketplace-specific attributes, or data from your PIM.
- Deduplicate by GTIN. Match products across suppliers and against your existing catalog by EAN/GTIN, not by title. Validate the GTIN check digit and normalize length first (more on that below).
- Apply price rules. Calculate selling prices from purchase prices with your margin rules: per supplier, per category or per brand, with minimum margins, rounding rules and price floors.
- Calculate stock. Combine supplier stock with your own warehouse stock, subtract a safety buffer, and decide what quantity each channel should see.
- Publish deltas. Compare the result with what was last sent and push only the changes to your ERP, shop and marketplaces.
Top tip
Keep the raw file, the normalized data and the published values for every run. When a customer reports a wrong price, you can trace in minutes whether the supplier sent it, a mapping produced it, or a price rule calculated it.
How do you match products from multiple suppliers?
When several suppliers offer the same product, you want one product in your catalog, not three listings that compete with each other.
The GTIN is the natural key. GTINs come in 8, 12, 13 and 14 digits (EAN-13 and UPC-A are the familiar ones), and the last digit is a check digit calculated with a modulo 10 method. In practice:
- Normalize to 14 digits by padding with leading zeros, so a 12-digit UPC and the matching 13-digit EAN compare as the same number.
- Reject invalid check digits instead of importing them. A wrong digit means a wrong product match.
- Watch for packaging levels. A GTIN for a case of 12 is not the same trade item as the single unit. Do not merge them.
- Keep a fallback for products without a GTIN, such as manufacturer name plus manufacturer part number, and review those matches manually.
Once products are matched, decide which supplier wins for each field. A common setup:
- Product content comes from the supplier with the best data quality, or from your PIM.
- Purchase price and stock are kept per supplier, so you know your cost from each source.
- The offer you publish uses rules, for example the cheapest supplier with stock, or a preferred supplier unless it is out of stock.
How do you detect changes and send only deltas?
Most supplier feeds are full exports: every product, every time. Pushing all of them to Shopify, WooCommerce, Shopware or a marketplace on every run wastes API calls and hits rate limits fast.
Instead:
- Hash each normalized record (or the relevant fields) and compare it with the hash from the previous run. Only changed records move on.
- Split fast and slow data. Stock and prices change often and are small; product content changes rarely and is large. Run them as separate jobs with separate schedules.
- Handle disappearing products explicitly. A product missing from today's feed may be discontinued, or the file may be incomplete. Mark it out of stock after a missing run or two instead of deleting it immediately.
- Run a full reconciliation on a schedule, for example nightly, to correct drift from missed updates. The same principle appears in our marketplace ERP integration checklist.
How should stock sync and buffers work?
Supplier stock is information about someone else's warehouse, and it is always somewhat out of date. Protect yourself:
- Use buffers. If a supplier reports 3 units, you might publish 0 or 1. Configure buffers per supplier, category or item, depending on how reliable the supplier is.
- Cap large quantities. Publishing "9,999 in stock" from a supplier feed rarely helps sales and can hurt if the number is wrong.
- Map stock texts. Some suppliers send "available", "low" or "on request" instead of numbers. Translate them into numbers your systems understand.
How do you handle errors and alerts?
A feed pipeline fails quietly unless you design it not to. At minimum:
- Alert when a feed is missing or stale, for example when a supplier's file has not changed for longer than usual.
- Alert on unusual changes: row count drops, a large share of prices changing at once, or prices outside sensible bounds.
- Quarantine bad rows instead of failing the whole import, and list them with a reason that someone reviews.
Common mistakes in supplier feed automation
- Importing without validation. One broken file can set thousands of prices to zero. Validate first, publish second.
- Matching products by title. Titles differ between suppliers and change over time. Use GTINs and an explicit mapping table.
- Hardcoding each supplier's format. Every supplier then becomes a code change. Use configurable mappings per supplier.
- Letting two systems own prices. If the feed pipeline and the ERP both set selling prices, they will overwrite each other. Decide who owns each field.
- Pushing full catalogs every run. It is slow, burns API quota and makes errors harder to trace.
- Trusting supplier stock as is. Without buffers, you will oversell.
- No owner. Suppliers change formats without warning. Someone needs to notice and fix the mapping.
Build, buy or outsource?
Feed management tools, PIM systems and shop plugins can handle the basics, and for a few suppliers with clean CSV files they may be all you need. A custom pipeline makes sense when you have many suppliers with messy or unusual formats, need custom price and stock logic, have to feed several systems at once, or your current tool keeps breaking on supplier changes. On the publishing side, the same pipeline usually feeds the marketplace integration that keeps listings on Amazon, Kaufland or eBay current.
FAQ
How often should supplier feeds be updated?
Match the supplier's own update frequency. Stock and prices are often worth fetching hourly or more often if the supplier updates them that often; product content is usually fine daily or weekly.
Can I automate suppliers that only send Excel files by email?
Yes. A dedicated mailbox receives the attachment, the pipeline extracts and validates it, and it then runs through the same steps as any other feed. The extra work is making the parser tolerant of small layout changes, and alerting when a change is too large.
What if a product has no EAN or GTIN?
Match it by manufacturer name and manufacturer part number, or by the supplier's SKU through a mapping table, and review those matches manually. Keep in mind that many marketplaces require a GTIN for most categories, with exemptions only in specific cases.
Do I need a PIM for supplier feed automation?
Not necessarily. A PIM helps when you enrich and manage a lot of product content. For stock and price updates, a feed pipeline connected directly to your ERP and shop is often enough.
Need help automating your supplier feeds?
We build and maintain supplier feed pipelines that fetch, clean and map supplier data and push it into ERPs, PIMs, Shopify, WooCommerce, Shopware and marketplaces. See our supplier feed integration service, or describe your suppliers and systems below and you will get a reply from the engineer who would build it.