MAG already had a mature stack of BI and database products for reporting. These were stretched and needed to be replaced by a solution that would create a scaleable and flexible data solution.  This would allow MAG to reach its ambitions.  The new solution would need to enable;

• an extended Data Warehouse
• scalable and elastic compute
• deal with seasonality spikes in passenger travel

This would allow real-time streaming of data; empowering MAG to become a real-time business across their entire customer journey.

This would give the control room and security staff a better view of the passenger flow and security performance. This allowed them to quickly make decisions based on real-time data. Data requirements included data ingestion from several on-premises systems and external data sources and then visualised on several large screens.

We addressed the first problem with a simple patch to the existing logstash-input-jdbc plugin. Before the patch, we could only express schedules in cron format. It used a Job Scheduler known as Rufus-scheduler, which supported supplying schedule expressions as either cron or number of seconds. The only change was a single line of code to use the repeat method instead of cron.

We started to use Kibana to visualise the data that was available in Elasticsearch hosted in Elastic Cloud. But we couldn’t quite do what we wanted. A more granular level of control was a requirement. We came across Canvas by chance, when the Head of BI from Manchester Airport Group and I were at the AWS Summit in London in May 2018. We spoke to the folks at the Elastic stand who mentioned Canvas and how they felt it was a better fit for what we wanted to achieve.

“We’re having difficulty in trying to visualise our data in Kibana as it doesn’t give the low-level control that we need”

“Have you seen Canvas?”

“No, what’s that?”

“Our new visualisation tool; it’s in technology preview at the moment, but it sounds like it might give you what you want”

Writing plugins for Canvas is very similar to writing plugins for Kibana. They are written in NodeJS and utilise registries which allow you to add additional functions and even new elements that can be selected inside the Canvas UI. We created 3 plugins to achieve the level of control needed.

We couldn’t use Markdown to control the style of text and images (though you can provide cascade style sheets within canvas, Markdown syntax does not have support for setting HTML class attributes for which style to apply).

The simplest approach was for us to create a plugin that would allow direct coding of HTML and in a similar way to the Markdown element, allow handlebars expressions for data binding.

As SVG images can be embedded directly inside HTML, we could apply the same level of control to dynamically change the images based on data retrieved from Elasticsearch.

It was time to take inspiration from the internet, searching for a “modern dashboard” on platforms such as Pinterest and Dribbble gave some interesting approaches to designing a dashboard and a different direction was taken.

The security hub dashboard again, it’s now a dark theme, with the use of gauges and colour gradients (still not finished, it will change more)

Several elements were built-in to the screen, mostly using the HTML plugin created. A mixture of simple HTML and some complex SVG images dynamically generated.

The initial feedback was positive, however, users that hadn’t been part of the design didn’t understand all the metrics without an explanation.

Wow! That looks a lot better, but what do the bottom of the bars mean?”

Going back to the original requirement of the Airport Security Operations being able to quickly make decisions, meant that although you could produce something that may look impressive, it’s not always the best way to visualise the information. Users instantly know what information the dashboard is displaying without it needing an explanation.

The following dashboards were the result of collaboration and creative thinking.

“The dashboards that synvert TCM (formerly Crimson Macaw) have created for us using Elasticsearch and Canvas have given Stansted Airport a modern real-time display of our data that was previously held in individual systems. This has created a number of efficiencies in our internal processes that would not have been possible before.”

– Stuart Hutson, Chief Technology Officer, Manchester Airport Group

Note, the data in all the dashboards is for presentational purposes only; using another Canvas plugin to do this!

In the first 6 months of the project, MAG went from a single instance database to a scalable Data Warehouse. Daily sales rung in at store level to over 90% of all sales automatically ingested from over 50 separate retailers.

By the end of the first year, MAG had added a real-time streaming solution to ingest data from the car park and fast-track bookings.  By the end of the second year, they had multiple real-time dashboards showing the state of the customer security queues. Data ingestion came from dozens of sources including trains, queues, metal detector scans, body scans, x-rays etc.