Technical Deep Dive: Jasper PIM Integration for BigCommerce Enterprise
At Jasper PIM, we focus our efforts on providing a seamless PIM integration for mid and enterprise technology stacks. We have clients like Skullcandy, Mad Dogg Athletics (formerly Spinning.com) and Avery Dennision, who publish their content not only to BigCommerce via our PIM integration, but also to third party professional search services such as Nextopia, Google Merchant Center (for remarketing ads), Amazon, and eBay. Having your staff master product only once and not countless times in multiple back-end admin consoles, is the only sane way to manage anything more than a few hundred SKU’s, let alone tens of thousands. A PIM really shines when integrated to your inventory management system (IMS), such as JDA, MS Dynamics Navision, NetSuite, etc. Many of these older legacy all-in-one back office platforms don’t offer an incredibly vast selection of product merchandising or meta-data capability on their own however, and usually fall down with respect to professional media management and provision for very fine grained customization. What’s more, a PIM enables sophisticated and extremely granular user access controls all centered around the product data itself, which becomes important in keeping your more junior staff away from pricing data in your ERP, or any other sensitive financial information in your accounting platform. You might not believe how many times I’ve seen junior staff with usernames/password access to the entire financial platform, including views of the balance sheet and income statement, just so they can also update web descriptions for their bosses’ online store! Another additional architectural nicety of a PIM is that you can train your staff on one solution to use, instead of having to train them on manually publishing product data to various storefronts one-by-one. Imagine the margin for user error here. In fact, when you’re done imagining, here’s a concrete and less imaginative stat to illustrate the point. The ratio for erroneous entry to accurate entry is typically 1:4. No wonder manually managed product data or (even worse, inventory counts) result in stock count fill issues and customer satisfaction & returns fallout rates of 25% or more.Publishing Products from Our Professional PIM to BigCommerce Enterprise
Now that I’ve detailed what a PIM is, and hopefully articulated some of the unadulterated sanity in using one, let’s outline below what was involved in connecting our PIM to the BigCommerce API. Our PIM was developed atop the increasingly popular Laravel MVC framework and the original prototype was developed specifically against the BigCommerce Product API in April of 2014. The first goal was to ensure that our PIM could support a significant number of SKU’s that would doubtlessly come from a professional ERP source or multiple inbound feed sources via automation. That necessitated in our case running creation tests against the BigCommerce API involving an estimated 150,000 SKU’s. Yes that’s correct… 150,000 SKU’s. That’s more than any one minimum wage product data entry clerk could ever hope to accurately manage manually – via the gorgeous and user friendly BigCommerce admin console no less – even if they had fifteen lifetimes to do it in. Automation here was key, and while we assumed the BigCommerce API was built for this sort of thing, we needed to run a battery of tests to corroborate the fact that the platform could stand up to our needs for timely and integral operation. Otherwise, we’d have been shopping around for another commerce platform.Speed Test
Adding tens or a few hundred products via the API we postulated would predictably take seconds or minutes. What would happen if we tried to add 150,000 products within the same hour? Our first attempt in using the Product API did take an unholy amount of time, unfortunately; on order of 3 days to complete, as we were also attempting to add as many as 35 custom product attributes to each product record. Product attributes are things such as color, size, SKU, series, brand, weight, height, condition, etc. We needed to get the ingestion time down to something more palatable and thus opted to package all of our attributes from our PIM into a single product entity field as a JSON object. Once we did that, we were able to get the creation time for 150,000 SKU’s down to a matter of 4 hours and 36 minutes. That’s not altogether shabby, since we weren’t planning to add 150,000 SKU’s every 15 minutes, or even every day. We just need to do this once when we onboard a new client during the initial load (or ingestion) process before a new store goes live. Since we predicted that most of our customers would likely only peak at about 30,000 SKU’s or so, we now have the initial ingestion time down to about an hour, which was totally reasonable.Integrity Test
At no point did we encounter any integrity issues with the BigCommerce API. An integrity issue is simply a case where we attempt to add or update a product by making a REST call to a function inside the BigCommerce API, and the function call either:- Leaves the data in a corrupt state
- Deadlocks or hangs the calling process
- Crashes the server or bungles up the store or database powering the API
Reset Test
The joy of using a PIM comes into play when you wish to make detailed wholesale changes to product content. At the time of writing, the BigCommerce API doesn’t have the provision for search and replace functions of product data, modifying attributes with special prefixes, adding watermarks to all product imagery, for example(s). All things a professional PIM could support, but would require a wholesale reset of the products inside BigCommerce. We tested a wholesale UPDATE and a wholesale DELETE. In both cases, the experiments produced satisfying results.
BigCommerce Support Kudos
Unfortunately, sometimes things go wrong. Once our PIM had been developed and approved by our client, things were going along quite nicely leading up until the planned launch. We did encounter however some last minute challenges in using the API that was a result of bugs (i.e. faults) in the PHP. However, the BigCommerce support team worked closely with us to pinpoint and resolve the issue in hours. This could have potentially delayed launch by days or even weeks. Sharing that story makes me want to take a moment to say that the BigCommerce support experienc,e from an enterprise integrator’s perspective, is that of legend. Having access to such an amazing team is essential to any solid project management planning efforts. My experience working with BigCommerce support was unparalleled in the industry, for the following reasons:- The support rep answered the phone quickly, was empathetic, and helpful.
- The support rep wasted no time in putting me directly in touch with someone more technical, who was himself both pleasant and astute; with a mastery of English that was refreshing. The rep was very patient as I described in painstaking detail the precise symptoms as best we could tell from our diagnostics.
- As soon as the rep realized he was out of his pay grade, he directed me to one of the engineer’s that was involved in mastering the API itself. Surely, if this individual didn’t know what the issue was, no one likely would.
- The API engineer provided clear direction and we iterated over the phone on a number of attempts to solve the problem, until the solution was at least sorted out.
Comments
