Cybersecurity data comes in from many different sources. While we have already discussed systems that provide (relatively) real-time protections, significant data from previous attacks suggests a delay of some weeks and months from breach to detection.

Using the whole suite of available tools and available data helps to improve our ability to detect threats, both in real time and retrospectively. To do this, information needs to be brought together in a common place and format. To improve searchability, and data integrity, normalization should be carried out. This reduces redundancy (hence becoming resource efficient) and helps to relate one log to another.

One key field for relating one entry with another is the time stamp, which can vary in format from one system to another. The other is the IP 5-tuple, which can be used to identify connections between endpoints, both...