The limits of a standard infrastructure recommended by Mautic
As proposed on Mautic’s open-source GitHub project, it is recommended to choose a LAMP stack infrastructure. We ruled out this option because performance would very quickly become limited as soon as we exceeded around ten clients, and this would have made our service unusable. We also hosted at Webmecanik the software of certain companies that were using Mautic Open Source with a substantial contact database, for example because they were very quickly reaching the limit of the infrastructure recommended by Mautic.
A horizontal scalability approach
To overcome this, we chose to pool resources from several VMs so as not to be limited by the capacities of a web server.
We therefore chose to:
- set up a load balancer
- boost the performance of the existing database for our clients with standard use of the software
- install a new high-performance database to which we moved the databases of our most resource-intensive clients
Today, our infrastructure consists of 6 application servers, including 5 high-availability VMs and 1 dedicated server, 2 file servers (filers), including 1 high-availability VM and 1 dedicated server, our load balancer, and our two databases. We are now in a position where installing a new application server is trivial.
As for the databases, they can still be scaled up before it becomes necessary to install new ones.
In other words, it is now easy for us to scale horizontally with each new client, and when it becomes necessary to install a new database, we will implement a database partitioning strategy that will allow us to easily distribute data across several servers.
We train our clients, and they use the software more and more
As a result of our clients making good use of the software, another challenge posed by the Mautic software was the distribution of scheduled tasks. Indeed, for each installed instance, it is necessary to set up several scheduled tasks to allow the updating of segments, campaigns, and the sending of emails.
The more clients use the platform, the longer the tasks become and the more of them there are to process.
On this new infrastructure, we could not let each application server execute these tasks in parallel, because that would have created a competitive situation. We therefore decided to set aside one of the web application servers to focus on the scheduled tasks of the entire infrastructure. We therefore set up a system that limited the maximum number of scheduled tasks running. This system allowed us to guarantee quality service under normal conditions and a reduced service that prevented a total system failure during periods of very high activity. This system served us well for a long time, but it depended on the sizing of a single machine and was therefore temporary.
We started working on a new system that would allow us to distribute these tasks across our entire infrastructure. Internally, we named this system: the sequencer.
The sequencer had to address many issues:
- How can a task be sent to the most available machine?
- How should tasks that fail be handled?
- And how can the same task be prevented from being executed on several machines at the same time?
The sequencer has recently been deployed on our infrastructure. Following the positive response of our infrastructure, we can accommodate more instances and even improve our performance.
