Imagine the eCommerce that:

  • can be deployed to any existing eCommerce platform (Magento2, Magento1, Shopware …) – so you can avoid all the integration you already have; focusing on the value for clients,
  • compliant with Progressive Web Apps standard,
  • renders the catalog of products within milliseconds,
  • queues orders if the server is unavailable,
  • works 100% offline without Internet access,
  • therefore is almost impossible to be crushed with huge traffic peaks/loads – it’s basically not using server resources to work.

The Proof of Concept

Sounds like something worth doing to me :) Especially after being a CTO for some time without the direct chance to code anything. I thought it may be a great journey!

A month ago we (I myself :), Karol Bzik and Filip Rakowski from Divante) just started 3 discovery projects that hopefully will bring us to this goal. At least we will publish some working MVPs:

  • vue-storefront – is the main frontend application – the project with no README.md yet ;),
  • mage2nosql – adapter between (currently) Magento and NoSQL databases,
  • vue-storefront-api – is the API part, providing the abstraction layer between backend platform (let’s say Magento2) and storefront; it uses the database that mage2nosql synchronizes.

We use Sketch to work with mockups/gfx – starting with Mobile First approach

We decided to go with PoC for Magento2 exclusivelyVue.js as frontend library, Node.js + Express (and maybe GraphQL support) as a server-API, Elastic Search as a database of products and full PWA/Offline support.

Oh, and it’s purely JavaScript.

Right now 2 of these 3 projects are on its way. I do not suggest you clone the storefront or API – because it’s not fully functional right now.

Service workers are important part of any PWA application. We’re just started – but here you are – service-worker is up and runing caching API calls, static files, making the whole app Offline friendly. Of course we’re using Vue.js SSR rendering as well.

Data pump

What can be useful is mage2nosql data pump. I had two initial goals starting the project:

  1. To deal with SQL database which can, and usually does, limit the performance,
  2. To create headless, platform agnostic API for modern eCommerce storefronts.

When You ran Magento (especially M1) on large catalogs (let’s say > 0.5M SKUs) You always had to deal with the database locks and all scalability stuff. When we had to deal with all this Redis/Memcached, Replication stuff – I was always wondering why the heck not to use simply the store which is a lot more scalable. Elasticsearch, MongoDB is extremely fast, horizontally scalable and easy to use.

Mage2nosql uses Magento2 API to synchronize all the data of products, categories and product-categories assignments to ES or MongoDB. It can queue the updates and update only the changed entities (using updated_at right now – then to be extended for checking the checksums or to use webhooks, which feature is already provided).

Cool thing is that, doing so (using API) you don’t have to change any single line within existing Magento2 deployment. The tool can run in the background and use multi-process – so even updating a huge number of products at once it’s pretty damn fast.

There you have the progress of the data sync. We’re using kue for Node.js to make workers run in parallel – even on many machines in the cluster at once.

The first run is simple.

1. Clone the master from https://github.com/DivanteLtd/mage2nosql

$ git clone https://github.com/DivanteLtd/mage2nosql.git

2. Enable your Magento2 API (System->Integrations->Add new integration). I’m using oauth2 authorization so you need the Acces Key, Consumer Key and so on.

Just update the ‘src/config.js’:

magento: {

  url: process.env.MAGENTO_URL || 'http://magento2.demo-1.divante.pl/rest/',

  consumerKey: process.env.MAGENTO_CONSUMER_KEY || 'alva6h6hku9qxrpfe02c2jalopx7od1q',

  consumerSecret: process.env.MAGENTO_CONSUMER_SECRET || '9tgfpgoojlx9tfy21b8kw7ssfu2aynpm',

  accessToken: process.env.MAGENTO_ACCESS_TOKEN || 'rw5w0si9imbu45h3m9hkyrfr4gjina8q',

  accessTokenSecret: process.env.MAGENTO_ACCESS_TOKEN_SECRET || '00y9dl4vpxgcef3gn5mntbxtylowjcc9',

 },

… or you can set the environment settings running the script, for example:

$ MAGENTO_URL=http://yourmagento.com/rest MAGENTO_CONSUMER_KEY=q5jcu7hoso96i1hdnnqivigqiie13wy2 
MAGENTO_CONSUMER_SECRET=o13jpyduwk6d995wyl39bdc0t1nhaox5 
MAGENTO_ACCESS_TOKEN=mpsvg7936erd8c5dfljx6ncgyg8wihwe 
MAGENTO_ACCESS_TOKEN_SECRET=0dn2ep56hsp9sfyx5ud6jxgcfdxoqgxc 
node --harmony cli.js products --partitions=1

3. To work with mage2nosql you need: MongoDB or Elasticsearch and Redis (because we’re using kue queue server with Redis backend). I’ve prepared docker config so it’s as simple as:

$ docker-compose up 

… to start Redis and ElasticSearch

Now you’re ready to run the tool.

$ node --harmony cli.js fullreindex

This command synchronizes all the categories, products and links between products and categories. First of all, it uses /categories endpoint on Magento2 API to get all the categories.

Then it gets all the category-product links and stores it in Redis cache (I didn’t find a simpler product-category relationship in API than /rest/categories/<ID>/products call for each category …)

Last but not least it gets all the products in 5 parallel processes and it uses result paging to not overwhelm your Magento server.

It uses kue queueing; one work item is 50 products (one page from API result).

You can track the results in kue-dashboard by running:

$ src/node_modules/kue/bin/kue-dashboard 

… and then opening http://localhost:3000 in your favorite browser

I’ve got some additional free time so did I implement additional commands for this awesome tool ;) You can find details in README.md

How do you find this tool? Is it useful for you? We’re searching for contributors, so don’t hesitate to make some commits :)

You can write to me directly: pkarwatka+storefront@divante.pl