Anomalies of Time and the LogPoint SIEM

"We trust that time is linear. That it proceeds eternally, uniformly. Into infinity. But the distinction between past, present and future is nothing but an illusion. Yesterday, today and tomorrow are not consecutive, they are connected in a never-ending circle. Everything is connected."

- Dark, 2017

Sometimes there are cases where search doesn't work, or rather, search "works" but it fails to return the logs we expect to be there. Due to these missing logs, we don't get to do our analysis and stuffs, dashboards are empty, alerts are not triggered, reports are empty, and existential crisis may start to kick in as we stare at our empty screens pointlessly. 

Well, there isn't a single answer to why this might happen. However, some of the most common yet subtle reasons often have to do something with time and its anomalies in Logpoint. By time, here we're mostly referring to the following terms: 

• log_ts : the time when a device generated the log
• col_ts :  the time when the log was collected in Logpoint
• index_ts :  the time when the log is stored and indexed inside Logpoint

Under different scenarios as discussed below, it's likely that the logs may be indexed and stored in some other time, past or future. These logs therefore don't qualify for the search timerange in our queries, hence the missing search results.

This introduction is likely very vague. To better understand this, let's jump straight into those different scenarios where logs might get teleported to a different time, and thus go "missing".

Possible scenarios 

1. Difference in the device's timezone and the timezone submitted in LP configuration
   • For instance, the device's timezone is actually UTC, but it's configured as Paris Time (CET, UTC+1). Here, the logs are stored 1 hour behind. 
2. Difference in timezone in the logs and in the LP configuration
   • Similar to point 1, however there might be services in a device that have their own service-specific timezone configuration i.e. the generated logs may not always take their time from the system clock. 
   • Hence, in this case, even if you've configured the correct device timezone in the Logpoint UI, there may still be a problem. 
3. The ISO date format: YYYY/MM/DD and the European date format: DD/MM/YYYY.
   • This issue is mostly observed with the normalizer services. The normalizers normally expect the logs to be in the ISO date format which is a widely accepted standard. However, if it encounters logs with a European date format, the Month and Day are interchanged. And, the logs you expect to be there on August 2 (08/02) will be stored on February 08 (02/08). 
4. SIEM's timezone configured to be something other than UTC in the UI (Configuration -> Devices -> Timezone) or in the backend (system clock itself)
   • In cases where the SIEM's timezone is configured to be something other than UTC, erratic behaviors might be expected. 
5. log_ts, col_ts and index_ts : delay leads to missing results
   • When we have a delay in the log ingestion pipeline due to persistent queueing, the log may get indexed significantly later than the time it's collected/generated (i.e. index_ts >> log_ts/col_ts). 
   • Also, if you've used fetcher services (e.g. Office365Fetcher, UniversalRestApiFetcher, etc) to fetch the logs only after a certain interval, a delay is expected in the collection i.e. col_ts >> log_ts. 
   • In such cases, the delay can lead to missing results in the searches. This is a very common reason behind alerts not getting triggered.
6. Device itself sending old logs or future logs
   • Sometimes, device itself might be sending old logs or future logs intentionally or unintentionally. 
     For e.g.
     
     • Intentionally, when old log files might be fetched using a file collection module with agents (e.g. Nxlog, LPAgent, AgentX, etc).
     • Unintentionally, when a faulty datetime is configured in the device itself.
7. Faulty normalization
   • Often, there are also cases where the datetime in the logs are wrongly normalized due to some bug in the normalization packages. 
8. Daylight saving time (DST)
   
   • Daylight Saving Time (DST) is the practice of setting the clocks forward one hour from standard time during the summer months, and back again in the fall, in order to make better use of natural daylight, as per this article. DST is implemented in many countries throughout the world, especially in Europe.

   • So, when the clock switch happens, one should be wary if their machines have been tuned accordingly as well.

   • Otherwise, one can expect to see an offset of 1 hour in the logs i.e. the logs are either an hour behind or an hour ahead depending on the time of the year.

How to identify? 

1. Check configuration on the devices' end.
   • Whether things be working good or not, it's always a good idea to recheck the configuration you have set on the devices' end, in this context particularly those configuration related to time. 
   • The following might be a good checklist:
     • The Timezone of the device 
     • The Timezone the device is configured inside Logpoint 
     • The Timezone in the logs
       • In some cases, the timezone in the logs and that of the devices may not be the same, as some entity may send logs in UTC standard by default. Different software entities configured inside the same device may send logs in different timezone.
     • The Datetime format in the logs
       • The preferred format of datetime might differ based on the region you're living as mentioned more clearly in this article. Whenever possible, it's always recommended to use the standard time format - ISO 8601.
     • Check if the timezone configuration in the device is DST-aware.
2. Sometimes by selecting a larger timerange for your queries, you may be able to find your missing logs and hence, identify the problem in hand. 
3. You can use the query | process diff(log_ts,col_ts)  to identify any delay between log generation and log collection. Similarly, any delay between log collection and the indexing of logs can be identified with the | process diff(col_ts,index_ts) . 
   
   E.g.
   
   
   References: process diff, process datetime_diff 
4. We might also get some indication from the logs of the normalizer service that logs older than 60 days or in the future by more than 1 day have been received.
   • E.g. 

   2020-08-30_01:03:19.69817 WARNING: LogTsDeviationChecker; Warning: Received log with timestamp older than 60 days; device_ip=10.45.1.10; norm_policy=windows; sample_log='<9>Aug 30 01:03:19 WIN-983F1HL8A07 Microsoft-Windows-Security-Auditing[0x4dc]: {"EventTime":"2019-05-27T06:50:43.314259-07:00","Hostname":"WIN-983F1HL8A07","Keywords":"9232379236109516800","EventType"'

   2020-08-30_01:03:19.69817 WARNING: LogTsDeviationChecker; Warning: Received log with timestamp in the future by more than 1 day; device_ip=10.45.2.20; norm_policy=windows; sample_log='<9>Sep 02 01:03:19 WIN-983F1HL8A07 Microsoft-Windows-Security-Auditing[0x4dc]: {"EventTime":"2020-09-02T06:50:43.314259+01:00","Hostname":"WIN-297F1JL8B09","Keywords":"342343516800","EventType"'

    

5. Look for any future or even unusual directories on the logs or indexes path. 

   • • Checking the directories in the storage path for logs/indexes path may sometimes help as well.
6. Tcpdump (the Support team way) 
   As someone from the support team, I personally find it easy to use tcpdump to check for traffic coming into port 5503 (the port that the logs get forwarded to after they're normalized) and check the log_ts value. Good thing is that you can run tcpdump as a li-admin user.
   
   An example is demonstrated below:
   
   
   
   

How to prevent?

1. Configure device's timezone properly in the log-sender side and Logpoint side.
2. Use consistent datetime format. It's always a good idea to use the ISO 8601 standard whenever possible.
3. Use CNDP plugin to adjust the datetime format (European or American) for the corresponding compiled normalizer. 
4. Do not tamper with SIEM's default timezone - UTC.
5. Please contact our Support team in cases where the time is normalized wrongly due to some possible bugs in normalization. 
6. Handle queueing as queue in the ingestion pipeline can lead to unexpected delays in log collection and indexing. Queue can mostly be identified by an unhealthy decline in the MPS inside the System Monitor. 
   
   Please contact our Support team in that case as well.
7. Delay alert parameter can be configured in cases where delayed logs are expected and the alerts need to consider those events before triggering incidents. 

Extra notes and tips: 

• Please note that if the timestamp in logs are older than the retention period, they'll be dropped and not be indexed. 
• As per your convenience, you can choose to perform searches based on log_ts or col_ts with System Settings -> Apply Time range on. It can particularly be useful in cases where delayed logs are expected. 
• You can also configure Over Scan Period (in minutes) and a Time Zone. The overscan period is time added to a log search, so that the scope of the time range for every search is expanded by the specified  number of minutes.
• In cases where you may need to renormalize the logs in searchtime with a different policy, you can use the norm command as explained in this article. 
• In extreme cases where you may need to reindex the logs that are already indexed incorrectly, you can request the Support team to perform the reindexing. Please note that it may be counted under Professional Support. 

So next time, when logs go missing and you might ask yourself how, maybe the correct question is when? 

"The question isn't how, the question is when." - Dark, 2017

References: 

• https://www.timeanddate.com/time/dst/about.html
• https://www.worlddata.info/timezones/daylightsavingtimes.php
• https://www.worlddata.info/timezones/date-time-notation.php
• https://docs.logpoint.com/docs/search-query-language/en/latest/Search%20Query%20Language/Process%20Commands.html#difference
• https://docs.logpoint.com/docs/search-query-language/en/latest/Search%20Query%20Language/Process%20Commands.html#datetimediff
• https://docs.logpoint.com/docs/cndp/en/latest/index.html#cndp
• https://docs.logpoint.com/docs/system-configuration/en/latest/System/System%20Monitor.html#system-monitor
• https://docs.logpoint.com/docs/alerts-and-incident/en/latest/Alert/Creating%20an%20Alert%20Rule.html#creating-an-alert-rule
• https://docs.logpoint.com/docs/system-configuration/en/latest/System/System%20Settings.html#general
• https://servicedesk.logpoint.com/hc/en-us/articles/115001514525-Re-normalizing-logs
• https://www.imdb.com/title/tt5753856/