Merging new updates into the registry


#1

We’re still working on and discussing the future of updating the registry without have to reset completely. At the moment, new entries into the registry should be added, but existing data doesn’t get overridden.

This has to do with the fact that, we have 3 ways to add data to the registry.

  1. in a plugin, through register.js
  2. in reaction.json in private/settings
  3. modify the database / do it thought the UI if available

So then, which one of those would you expect to be the source of truth? Which one overrides the other, when you don’t know which one was updated last. This is a major point of confusion and frustration for new and veteran reaction developers.


How it works now, and why

At the moment the settings and data in the database is considered the truth for the App. While your app is running in production. Any fixture data is to be assumed to be a one time deal to set up that app from a fresh install. Fixture data is not used as a source while the app is running, only the values stored in the database. You can consider the fixture data to be the defaults for your app when you do a clean install.

In development, the roles fell like they should to be reversed. While testing you app you may edit registry entries in the register.json files to see changes in real time; however, since this data is loaded only once, you’ll never see the change reflected anywhere in the app or database. This forces you do do a full reset every time you make a simple change, which is very time consuming.


Proposed Solution

Add the ability to selectively or globally, reset to data from your fixture files. Keep backups of settings to allow for reverting broken setting updates. This way, you get the best of both worlds. The ability update your local fixture data files and have the changes overwrite your database if you choose to.

Heres a diagram depicting this:


#2

I think we talked about using this package to handle migrations. It allows for forward/backwards migrations from the command line.


#3

Better Image:


#4

Thanks for getting this discussion started, I think this is a pretty important feature and it will be good to have some conversations about it.

From my perspective, I’d break this up into a few different sections

  1. Fixture data that is necessary for the app to load the first time.

  2. Layouts, workflows, routes, permissions, and other registry items.

  3. App data

I don’t know if there’s a good one-size fits all approach.
Here are some thoughts on the three different types of data and migrations that might be necessary:

Fixture Data

Fixture data could be replaced by a good admin onboarding process for production apps, or any apps that don’t have fixture files. Wordpress does a reasonable job of this, but I think a good onboarding process could serve as a product tour and a way to get the initial data setup correctly. Ideally the shop owner could be prompted for vital information and that information stored in the db. Shop name, description, SEO/Meta, Tags, API Keys (payment gateways, email, analytics, etc), locale and currency selection, shipping, and creation of a first product could all be part of a good onboarding process. Ideally this could eliminate the need for user editable fixtures and the conflicts that come with having two sources of data.

Layouts, Workflows, Routes, Permissions, and other Registry Data

This is where the biggest need is currently in my opinion. There’s not currently a smooth way to change registry data after making changes to a plugin without either wiping the whole DB.

Adding routes to an application seems to be a standard need, we’ve added routes a number of times since launching, and every time it’s necessitated manual updates to the production database. Permissions are currently connected pretty tightly to routes and can cause problems as well, especially adding routes that default users need to have access to. This is another topic, but we recently added a few static pages that needed to be accessible by all users. Adding access for anonymous users wasn’t too big of a challenge (db change), but adding access to existing users proved to be more work than it was worth as it required changing 20k+ user accounts to add permission.

Layouts and Workflows also need to be able to be updated on the fly without a db dive or reset and I think the use case here is pretty clear as well. Something like changing the template that a specific routes uses, or changing the workflow of the cart might need to happen on a production server.

I’m not sure the best way to deal with this, as some workflows will almost certainly conflict with others, especially if you’re changing anything that’s defined in core RC, but I do think it’s a most important need currently.

App Data

IMHO this should never get touched unless specifically requested. It also feels like a dev only conversation as you’d never consider nuking your production app data.

It’s annoying to have to re-seed app data after running ./reaction reset in dev, and not everyone keeps their test data in fixtures. Most of our data is in excel sheets that we built a custom uploader for, but we also have lots of shipping data, test orders, etc that become a pain to add to the database every time we reset.

Being able to reset all of the packages, settings, layouts, workflows, routes etc without losing app data (products, orders, etc) would be valuable.

Kinda rushed this post out as I’ve got to run, but hopefully it makes sense.


#5

So I don’t know if you have already looked at this (my feeling is that you probably have), but couldn’t you implement registry changes using the percolate package? At least as a stop gap until Reaction integrates that into our process? Maybe we could brainstorm on some method where we could auto-generate migrations from changes (this is the method I have seen most used).

This would at least go a long way to mitigating some of the issues with registry (but not permissions). That should probably be an enhancement to get these dynamically from the DB rather than a record on the user. IMHO


#6

Hi,
I am facing this migration issue now.
Percolate looks like a good option if migrations are autogenerated.

Can anyone tell how they maintain prodction upgrades now?