Hi All,

We are excited to share the latest blog post by Logpoint Security Researcher, Bhabesh Raj Rai. In this blog, we explain how Logpoint analysts use randomness calculations through the entropy plugin to create another layer of defense against attackers. To read the full article please follow the link below:
https://www.logpoint.com/en/blog/embracing-randomness-to-detect-threats-through-entropy/

Hi All,

In our latest blog, Logpoint Security Research blackbelt, Rabindra Dev Bhatta guides you through the steps to analyse, and mitigate HIVE ransomware, which as of November 2022 has targeted more than 1,300 businesses globally and have bagged almost US$100 million in ransom.

To read the full article, please follow the link below:

https://www.logpoint.com/en/blog/tools-and-tactics-to-track-down-hive-ransomware/

Hi All,

We are excited to share our latest blog written by Logpoint Security Researcher, Anish Bogati this time about Royal Ransomware, and the means to detect, investigate and respond to it with the help of Logpoint.

To read the full article, please follow the link below:

https://www.logpoint.com/en/blog/exploring-the-exploit-of-royal-ransomware/

We are excited to announce the date for our ThinkIn 2023 conference for global Logpoint users and partners. Here’s some key info:

Copenhagen, March 7-8, 2023 @Crowne Plaza Copenhagen Towers

As we are currently working on the official agenda, we encourage you to share your ideas on what kind of workshops and keynotes you would like to participate in.

All ideas are welcome in the comment section below or via email to customersuccess@logpoint.com .

Participation is free of charge. So all you have to do is get to Copenhagen and find a place to stay. Simple!

We are looking forward to seeing you.

Click here for pre-registration

Hi All,

SpoolFool, also known as CVE-2022-21999, is a local privilege escalation flaw in Microsoft Windows’ print spooler service, which controls print operations. Read Logpoint´s take on it along with detailed steps to protect your organisation in the article below, written by Nilaa Maharjan.

https://www.logpoint.com/en/blog/a-spools-gold-cve-2022-21999-yet-another-windows-print-spooler-privilege-escalation-2/

Hi All,

I would like to draw your attention to a recently released KB article discussing things that need to be looked into if the connection from the collector to backend server isn't being created while working with distributed Logpoints.

To read the full article, please follow the link below:

https://servicedesk.logpoint.com/hc/en-us/articles/5139550813213-Connection-issues-with-distributed-collector-in-logpoint

Read latest blogpost where LogPoint provides an overview of the research conducted on the emerging SpringShell family of vulnerabilities – A Remote Code Execution(RCE) based exploit found in the popular JAVA framework, Spring. The blog post is accompanied by a Logpoint Emerging Threats Protection report, covering detection methods, investigation playbooks, and recommended responses and best practices.

Enjoy

https://www.logpoint.com/en/blog/springshell-not-spring4shell-a-blooming-hype/

/ Brian

LogPoint stores its internal logs in the /opt/immune/var/log/ directory. Here is the description of the most useful log types inside that directory.

1. Service logs
   /opt/immune/var/log/service/<service_name>/current
   This holds the internal service logs for each of the service of LogPoint. The current is the file in which logs are currently being written. The log files are regularly rolled over and are kept in the same directory as current .
   
2. Benchmarker Logs
   /opt/immune/var/log/benchmarker/<service_name>.log
   This contains the benchmarker data for various services. Not all services are benchmarked but important services like norm_front, normalizers, storehandlers, indexsearchers, premerger, various collector & fetchers have benchmarker data in this location.
   

3. System Metrics Logs
   /opt/immune/var/log/system_metrics/system_metrics.log
   This contains the operating system metrics that is regularly benchmarked in the system. It contains the information of CPU Usage, Memory Usage, Network Usage, Hard Disk Usage kept every 1 minute.
   

4. Audit Logs
   /opt/immune/var/log/audit/<service_name>.log
   This contains the audit logs of various services in LogPoint. For example, one of the useful is that of webserver, which contains information of which user made what changes at what time from the UI.

These information can also be obtained from LogPoint's Search as these logs are collected using the file system collector into the _logpoint repo. These are normalized as well, so with appropriate queries we can even construct dashboard widgets and alert rules based on the data from these internal log files.
Some useful queries:

| chart count() by source_name

label = Benchmarker

"source_name"="/opt/immune/var/log/benchmarker/norm_front.log"| timechart avg(actual_mps)

"source_name"="/opt/immune/var/log/benchmarker/store_handler.log"| timechart avg(actual_mps)

We occasionally encounter cases where we cannot perform search because a particular repo has failed. In that situation, the search UI does not allow to make any searches if that repo is selected.

What does "repo failed" mean?

The "readiness" status of service responsible for searching in a particular repo (indexsearcher) is kept by central searching service (merger) in "alive" field for each repo. If the "alive" status of a repo is false in the config file of merger service, failed repo issue is seen when making search in that repo. This could happen if index searcher service for that repo is not running as expected or if the config file of the merger service is not updated according to the status of indexsearcher service.

Mitigation:

Whenever we get repo failed for a particular repo then it is always wise to check the logs for the indexsearcher service of that repo.

tail -f -n 50 /opt/immune/var/log/service/indexsearcher_<repo_name>/current#replace <repo_name> with actual repo name. for e.g. for repo with name "Windows",the command will betail -f -n 50 /opt/immune/var/log/service/indexsearcher_Windows/current

The above command will output the last 50 logs for the indexsearcher service of the particular repo.

You can also check if indexsearcher is replying back to alive probe with tcpdump on query.source.socket.

grep "query.source.socket" /opt/immune/etc/config/indexsearcher_<repo_name>/config.jsontcpdump -i any port <query.source.socket port> -Aq

If indexsearcher is alive, you should see {"isalive":true} and {"alive":true} messages

If there are no errors in the tail command and "alive":true messages are being seen in tcpdump commands but the failed repo error is still being seen with search,  try checking alive status in merger config.

grep -B1 alive /opt/immune/etc/config/merger/config.json

Potential Scenarios

1. Indexsearcher service is recently restarted
   If an indexsearcher service for a repo is just restarted then for large repos it takes few minutes to scan metadata for stored indexes, before searches can be served. During that period, the repo failed error is observed.
2. LogPoint machine is recently rebooted
   If a LogPoint machine is recently rebooted then the indexsearcher services take time to initialize services. During those few minutes, repo failed error can be seen for some repos.
   
3. Issue in the indexsearcher service
   If there is some error in the indexsearcher service, then repo failed issue does not resolve on its own.
   In such scenarios, please review the logs of the indexsearcher service as mentioned above. It is recommended to create a support ticket in such scenarios for further investigation and resolution of the problem. It will be helpful to include the service log of indexsearcher service of that repo in the ticket. The log file is located at

   /opt/immune/var/log/service/indexsearcher_<repo_name>/current

When we encounter cases when report fails to generate, There might be several reasons why this happened. Here are the list of things that can be attempted to handle report generation issues:

1. Try running the report again
   Sometimes, report might fail when there is high load in the system. And just trying to run the report again after sometime might result successful generation of report.
   
2. Try increasing the heap_size of the report
   There is an option in the report template (in the right side) to increase the heap_size of the report job. Large reports (reports with more number of queries and large time ranges) require larger memory.
   So, increasing the heap_size incrementally from 512 MB to few GBs (up to 4GB) can be helpful.
   
3. Try searching for each of the query in the report in search, if they are working fine or not
   This is kind of a brute force way to figure out the problematic query or query which might be causing the report failure.
   If any of the repo is down or if any of the query has issues, then report might fail. So, even though this step might take some time, this can be really helpful.
   
4. Try splitting the report if there are too many queries
   Sometimes, due to large size of the report (both in number of queries as well as in terms of timerange), the amount of data to be handled can be rocket high, hundreds of millions of events.
   Splitting the report both in terms of queries as well as splitting the timerange of the report can also be helpful.
   
5. For Scheduled Reports
   If the report is scheduled, it is wise to see if there are any other reports scheduled at the same time or not. If that is the case then, trying to schedule less reports simultaneously can help. This will reduce the load in the system.
   
6. Contact to support
   Even after trying above steps, if the report is still failing then it's time for some expert eyes. Please contact the support in that scenario. But try to attach the following information in the beginning itself so that support staff can act quickly in the issue.
   • Attach the screenshot of the error message as seen in the report jobs status.
   • Also, attach the following file from the backend: /opt/immune/var/log/report/report.log
     This file contains more details about the failed report.

Some Potential Scenarios

1. Report Gets Timed Out
Sometimes large reports might take more time to complete than the default time out period for the reports (30 minutes). If the report times out, the timeout period needs to be increased to accommodate for large reports. This can be done by support staff.

2. Report stays at running state and neither completes nor fails
If the indexsearcher service is restarted midway when a report is generating, then the report can stay in running state. In that case, we need to check if any related repo's indexsearcher service has restarted when report was running.
And if that is true, cancel the report job and start generating report again.

Occasionally, we encounter issues where one of the mount point in LogPoint is full. The disk full conditions can be because of various reasons and it is of utmost importance to immediately free some disk so that LogPoint can function normally.

Under disk full situations for /opt and primary repo path /opt/immune/storage locations, the log collection of LogPoint will be affected.

Detection

To detect the disk full situations, we can use the df command.

df -h

In the output of this command, we can either look for the percent usage of each mount point or the available storage space. These indicators will help us detect disk full scenarios.

Mitigation

Now, once we find out the problem in LogPoint is because of lack of storage space, we can dive deeper.

/opt path has 100% storage used

The /opt mount point generally stores the necessary config files, service log files, mongoDB data and executables of LogPoint. For normal functioning of LogPoint it is critical to have some storage space available.
Since in normal scenario, this mount point does not actively store much data, it is unlikely to have storage space 100% used. But when it encounters such cases, we have to investigate using du command and find out which directory or file is the cause of disk getting full. The command that helps out is as follows:

du -sch <file_1> <file_2> <directory_1> <directory_2>#To check all files and folder in current working directorydu -sch *

It is important to try this command manually across the directories inside /opt to detect the culprit. Note: /opt/immune/storage is usually mounted to a different pool or lvm.

Frequently encountered cases

1. Storage occupied by old upgrade patch files.
   The old upgrade patch files are stored in /opt/immune/var/patches/installed/ directory. These patch files range from few MBs to hundreds of MBs and they can be reason for /opt being full. These older patch files can be deleted, if we are sure that these old upgrades are successfully installed in LogPoint.
   
2. Storage occupied by mongoDB data directory
   The mongodb's data directory is /opt/immune/db/ . Sometimes the size of the db can be huge when the LogPoint has too many configurations data.
   In that case, please contact LogPoint support.
   
3. Storage occupied by service log files
   The service log files are stored in /opt/immune/var/log/ directory. In some cases when some service is in debug mode or due to some errors some log files can swell to unexpected size. In that cases, we have no option but to delete such files. We have to locate such anomalous files and delete them. This can be done by the same du command to check file size.
   Since the content of those files are already indexed into LogPoint's log ingestion pipeline it is fine to delete the service logs. But only do so, if you are sure, else contact LogPoint support to do so.
   
4. Storage occupied by nxlog dump files
   We have observed this issue in few customers when nxlog dumps some files in the directory
   /opt/nxlog/var/spool/nxlog/ .
   These files might can cause storage full in /opt mount point. So, cleaning the dump files or just moving them to other larger mounts should help. This issue has been addressed by recent version of LPAgent so, please update it to latest one to avoid having this issue.
   

/opt/immune/storage has 100% storage used

Usually /opt/immune/storage mount point has larger storage space compared to /opt because it has to store the logs and indices files as primary retention path.

If this mount point gets 100% used, then log collection gets halted and related services will stop to function. It is important to fix such issues. To drill down which directory might be using a lot of space, same old du command does the trick.

The probable cases when /opt/immune/storage is full can be as follows:

1. Storage occupied by logs and indices
   In most of the cases, when /opt/immune/storage is full, this is because of the logs and indices. The logs and indices directory grow in size because of the data stored by LogPoint.
   
   In normal scenario we would expect disk size to be estimated properly so that, the logs stored will not exceed the provisioned space. Sometimes for some repos however there might be abrupt increase in event rate. In such scenarios we can either decrease the retention for the repos with most amount of data. Otherwise, we need to allocate more disk to accommodate increased log volume.
2. Storage occupied by buffer directories
   There are some buffer directories which sometime can fill up, due to issues in the LogPoint and that can cause storage full scenarios. These buffer directories can be as follows:
   • /opt/immune/storage/highavailability/ - Issue in the highavailability (HA) functionality.
   • /opt/immune/storage/OldLogsKeeper/ - There are too much old logs coming in to the LogPoint machine.
   • /opt/immune/storage/FileKeepeer - If there is an issue in the indexsearcher service then logs are buffered in this directory.
     
     If any of the above directory are occupying too large space, then please call support for assistance.

In any of the above situations if you are not sure, it is important to call support for help. The paths mentioned here are for default installations. For some custom changes in the data mount point and so on, the paths might differ.

Note : The paths /opt/makalu and /opt/immune paths are actually same because in Logpoint /opt/immune is a soft symlink to /opt/makalu .

Please follow the documentation for LPAgent at https://docs.logpoint.com/docs/logpoint-agent/en/latest/index.html carefully. Sometimes simple steps like processing policy or even the collector/fetcher are missed while configuring devices.

Important Considerations for LPAgent:

https://docs.logpoint.com/docs/logpoint-agent/en/latest/Important%20Considerations.html

Logs should be coming into the system in few minutes once the setup is done, but sometimes we don't see any logs in the search. Below we have described few common issues faced on customer environment and their debugging procedures.

Service Status:

Check if lpagent process is running on windows side & lpa_manager and lpanx_collector services are running on LogPoint.

> netstat -nap | egrep ':7998|:7999'____________________________________________________________tcp 0 0 0.0.0.0:7999 0.0.0.0:* LISTEN 30445/lpanx_collecttcp 0 0 10.10.1.74:7999 10.10.1.67:50459 ESTABLISHED 30445/lpanx_collect tcp 0 0 10.10.1.74:7998 10.10.1.6:64664 ESTABLISHED 30445/lpanx_collect tcp 0 0 10.10.1.74:7999 10.10.1.65:49956 ESTABLISHED 30445/lpanx_collect# 7998 or 7999 should be on listening state and # various established session should be seen with windows ip

LPAgent Issue:

1. There is a bug on lpagent installation for 6.11.x+ LP where necessary dependencies aren't installed properly. We can verify it by running below mentioned cmd. The fix should be released on future version but current workaround is to reinstall the plugin, make sure normalization packages and lpagent collector are removed from devices before deleting the application.

   > tail -f /opt/immune/var/log/service/lpanx_collector/current____________________________________________________________FileNotFoundError: [Errno 2] No such file or directory: '/opt/nxlog/bin/lpanx_collector'

2. Nxlog has deprecated batching, so if you have enabled batching mode, please disable it and restart the lpa_manager and lpanx_collector service.
3. Orphan or duplicate process of lpanx_collector running, only one instance should be running on the server

   > ps -ef | grep /opt/nxlog/etc/lpanx_collector.conf | grep -v grep____________________________________________________________loginsp+ 30445 30404  0  2021 ?  04:22:00 /opt/nxlog/bin/lpanx_collector -f -c /opt/nxlog/etc/lpanx_collector.conf

Config & Cert Discrepancy:

If the windows side has only encrypted port allowed and logpoint agent is re-installed or cert is regenerated, windows won't accept connection with new certs. In such cases we can manually copy the conf and certs to the windows side.

Doing manual changes on large number of servers can be hassle. Therefore for syncing all the severs automatically following steps are recommended:

• Remove logpoint agent and associated device from configuration
• Re-install lpagent application on logpoint and allow 7994 port in windows side
• Add a new device and set the connection on TCP mode
• Once the connection is done, change the mode to TLS and push the config

File Permissions and Types:

Sometimes event logs are received but file collection module isn’t receiving any logs, either the file path wasn’t set correctly or the lpagent didn’t had enough permissions to read those files on the system.

Evtx files can't be fetched using file collection module.

Queue in Normalization:

Windows logs can cause queue in normalization layer if the normalization packages are in wrong sequence. For example; always make sure windows normalization policy has LPA_Windows above DNSCompiledNormalizer (has some regex checks which takes longer time). After that restart the normalizer service and queue should be decreased, 2nd and 3rd column would have persistent large values if queue is present in below output.

netstat -nap | grep :5505____________________________________________________________tcp        0      0 127.0.0.1:5505          0.0.0.0:*               LISTEN      -                  tcp        0      0 127.0.0.1:58306         127.0.0.1:5505          ESTABLISHED -                  tcp        0      0 127.0.0.1:58092         127.0.0.1:5505          ESTABLISHED -                   tcp        0      0 127.0.0.1:5505          127.0.0.1:58292         ESTABLISHED -                  tcp        0      0 127.0.0.1:5505          127.0.0.1:58284         ESTABLISHED -                  tcp        0      0 127.0.0.1:58284         127.0.0.1:5505          ESTABLISHED –

Connection Issue:

Firewall Rules:

Port 7994-5 is used by LP to share conf and certs with windows. Inbound firewall for 7994 and 7995 needs to be open at windows.

[7994] – Unencrypted Communication Channel[7995] – Encrypted Communication Channel

Port 7998-9 is used to collect logs from windows by a service called lpanx_collector, outbound firewall is needed for 7998-7999 from windows to LP

[7998] – lpanx_collector [Unencrypted data channel][7999] – lpanx_collector [Encrypted data channel]

Note: This rules applies to the Collector LP Machine not Backend, if backend and collector LP are different. Please make sure the firewall or switch between logpoint and windows also have necessary rules and routes setup.

How to test if the endpoints are reachable and connected from Logpoint:

For unencrypted connection:

> telnet <windows_ip> 7994____________________________________________________________Trying windows_ip ...Connected to windows_ip# the windows is reachable from logpoint side> telnet <windows_ip> 7994____________________________________________________________Trying windows_ip ...telnet: Unable to connect to remote host: Connection refused# the windows is not reachable from logpoint side

For encrypted connection:

> openssl s_client -connect <windows_ip>:7995____________________________________________________________CONNECTED(0000003) # if reachable and certs are correct____________________________________________________________connect: Connection refusedconnect:errno=111 # if unreachable or certs are invalid

Current status:

> netstat -nap | egrep ":7999|:7998" | grep ESTABLISHED# Output contains all the established connections > netstat -nap | egrep ":7999|:7998" | grep -v ESTABLISHED# Output contains all the connections that logpoint is trying to make but fails# 5th column is the windows ip

On windows side you can just check the logs files to see if the connection is being broken or refused at C:/Program Files/lpagent/data/lpagent.log

CPU/Memory Spike:

• Unsupported file formats can cause overhead on nxlog processor.
• Selecting mem buffer instead of disk buffer can also cause memory spike if the connection isn't stable.
• Older LPAgent version had a memory leak, so please upgrade the plugin to latest version.

Note: LogPoint Agent is powered by NxLog and is just a user friendly way of sending logs via windows. If you are comfortable with nxlog, it can directly send logs to syslog port of LP which would eradicate some of the issue with LPAgent. For example: collection of custom evtx files.

Some of the commands might require root access, please consult support if above mentioned steps are confusing or didn't help you figure out the cause for the issue.

CVE Summary : A Spring MVC or Spring WebFlux application running on JDK 9+ may be vulnerable to remote code execution (RCE) via data binding.

CVE Reference : https://nvd.nist.gov/vuln/detail/CVE-2022-22965

Logpoint Summary :

LogPoint uses spring framework in report engine, to connect with MongoDB using Spring-Mongo api. So, as transitive dependencies, spring-beans-3.1.2.RELEASE.jar is used. LogPoint does not have external exposure with this spring library as we are not using controller and accepting web requests via spring framework. Thus, we are not directly impacted by this vulnerability. We are also safe because of the following:

• We are not using any spring web framework.
• We are using Java 8 not Java 9 and beyond.
• We are not packaging in WAR and not using any servlet containers.

Director

• For LPSM, the impact is the same as for LogPoint.
• For other components in Director, we are not using spring libraries.
• We are still using Java 8 all across the product line.

Soar

• Currently any SOAR container running Java are running on OpenJDK 8 Alpine base containers.

The predictable naming scheme on interface is disabled due to the line below in grub configuration

GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

as a result the system with more than one network interface cards can change for example eth0 can become eth5 or eth3 can become eth2 and so on across reboots.

Reference for getting back legacy scheme here .

Note: Following activity requires sudo access/run as root user.

Inside  /etc/udev/rules.d/70-persistent-net.rules, change the entry like this with proper mac address:-
Include all physical network interfaces:- for example(assuming the system as 6 interfaces). Here eth0 with mac address 28:05:Ca:d8:fc:1a has active link.

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:05:Ca:d8:fc:1a", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="31:e1:71:6b:68:ad", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="ether1"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="31:a1:71:6b:68:ae", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="ether2"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="30:b1:71:6b:68:af", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="ether3"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="28:cf:37:0d:0e:c0", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="ether4"SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="18:df:37:0d:0e:38", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="ether5"

Create a symlink:-ln -s /dev/null /etc/systemd/network/99-default.link

udevadm triggerudevadm control --reloadupdate-initramfs -u

Reboot the system to see if the interface name is set as expected.

In case of LACP bonding simply use etherN name for all interfaces, reboot and run ethbonding command to configure bond interfaces.

To verify bonding is working or not you can ssh to the machine & run the command below

cat /proc/net/bonding/bondN  

unplug one of the cable and see if ssh connection is still working.

Another approach (for non-bonded setup )beside above would be only keeping one interface and detaching excess interfaces from the system.

Process ti is used in Logpoint to use the search time enrichment functionality from the threat intelligence database. This can be done by using | process ti(field_name) in the query.

Example:

source_address=* | process ti(source_address)  

This query will enrich all the logs which match the source_address field with the ip_address column present in the threat_intelligence table. This source_address to ip_address (and other) mappings can be configured from the threat intelligence plugin's configuration settings.

The search query above takes a large amount of time to complete if there are a large number of logs present within the search time frame. We can optimize queries by filtering logs before piping to process ti() part in the query.

Example:

norm_id=* source_address=* | process ti(destination_address)

This query can be optimized by using another query:

norm_id=* source_address=* destination_address=* | process ti(destination_address)

This query will select only the logs that have destination_address in them, as it would make no sense to match logs that do not have destination_address, using destination_address with threat intelligence database.

Although this does some optimization, for a large number of search results this query also takes time and hence further optimization can be done to complete this query in a small amount of time.

For this optimization, first, we need to store the corresponding column from threat_intelligence table which needs to be matched with the actual query result into a dynamic list. For example:-  If we need to map IP field like source_address or destination_address, we can run the following query to populate a dynamic List (e.g MAN_TI List) with ip_address field of threat intelligence database.

Table "threat_intelligence" | chart count() by ip_address  | process toList("MAN_TI", ip_address) 

Similarly, for another field, we can do

Table "threat_intelligence" | chart count() by field_name | process toList("Table_name", field_name)

Once we have populated data in the dynamic list we don't need to populate this until the age_limit set on Dynamic list(MAN_TI)

After the table is populated we can change the query

norm_id=* source_address=* destination_address=* | process ti(destination_address)

to

norm_id=* destination_address in MAN_TI  | process ti(destination_address)

In generic form, an optimized query would be

Filter_query field_name in Dynamic_list | process ti(map_field_name)

This query will complete much faster than the original query.

Note: If there is a large number of entries in the created dynamic_list created above then we will see some error in UI reporting max_boolean_clause count reached. This would mean we need to tune a parameter in Index-Searcher / Merger / Premerger services to increase the number of boolean clauses allowed.

Parameter_name: MaxClauseCount

We need to increase this configurable parameter based on the number of rows present in the dynamic list.

This can be configured in lp_services_config.json located at /opt/immune/storage as follows:

{"merger":{"MaxClauseCount":<value>},"premerger":{"MaxClauseCount":<value>},"index_searcher":{"MaxClauseCount":<value>},}

After this change, we need to regenerate the config file with the following command:

/opt/immune/bin/lido /opt/immune/installed/config-updater/apps/config-updater/regenerateall.sh

Try increasing the value of MaxBooleanCount continuously until the UI search does not throw an error and the search completes smoothly. A good rule of thumb would be to start with double the number of entries in the dynamic list.

This optimization can have some repercussions as well:

In the case of search_head-data_node configuration if bandwidth between these two instances is low, then we can get issues in the search pipeline as the query that needs to be transferred from search_head to data_node would increase in size after this optimization. If the bandwidth is not very low, then this optimization will not have such a significant impact on the system and it would increase search performance significantly.

I addition to the blog-post about LogPoint and Log4j update here: https://www.logpoint.com/en/blog/status-on-log4j-vulnerability-in-logpoint/ - LogPoint also created another Blog-Post on how a combination of tools and a defense-in-depth mindset will give organizations the ability to detect post-compromise activity and put a stop to the attack

Read it here: https://www.logpoint.com/en/blog/detecting-log4shell-requires-more-than-just-a-siem/

Regards,

Brian Hansen, LogPoint

VP, Customer Success

→ I’m not responsible if anyone Brick the System! ←

For everyone who is using Rapid7 InsightVM:

We upgraded the provided LogPoint Script with the ability to import SITE Scans instead of normal Scan Reports.

The Reason is that SITE Scans are scheduled on the IVM so the complete Vulnerability Scan and Import into LogPoint is without any manual interaction - there is no need to run the Report manually.

What to do on LogPoint: Just replace the script attached here with the Script installed on your DataNode and configure the Rapid7 integration.

What to do on Rapid7 Machine:

1. Create user with following permissions:

- Appear on Ticket and Report Lists: Appear on user lists in order to be assigned remediation tickets and view reports.

- View Site Asset Data: View discovered information about all assets in accessible sites, including IP addresses, installed software, and vulnerabilities.

- Create Reports: Create and own reports for accessible assets. Configure all attributes of owned reports, except for user access.

- Use Restricted Report Sections: Create report templates with restricted sections. Configure reports to use templates with restricted sections.

- Manage Report Access: Grant and remove user access to owned reports.

2. Create Report for Sites/AssetGroup/Assets/Tags/Scan

- Set Name: LogPoint_XML_Site_Report

- Set ReportTemplate: XML Export 2.0

- Select Scope: Site_ABC

- (Optional) Use only assets found in the last scan

- Frequency: Run a recurring report after every scan

- Save & Run the Report

Please find the material presented during the 2021 ThinkIn session on Alerts and Widgets attached to this article as well as the complete video recording from the session on Youtube via this link:

https://www.youtube.com/watch?v=I2OvY72XtrY

The scripts mentioned during the presentation can also be downloaded from here.

A few words on each of the scripts.

pm_analyzer.py

The primary purpose of the pm_analyzer.py script is to provide information about the performance of each livesearch. It is usually run interactively from the shell on systems experiencing performance issues with livesearches - widgets not updating, or alerts not firing as they should.

Although the script will need to run for as long as the longest update cycle for any livesearch takes (which it doesn't inform about - yet) to provide complete information on all livesearches - on most systems the issues would probably become evident sooner as the performance would most likely be impacted by resource hogs that are updating frequently and therefore become visible faster.

The script uses tcpdump on the merger port lo:5555 port and therefore requires root privileges to run.

premerger_mon.py

This script will read the premerger configuration and based on that information monitor the merger port lo:5555 to verify that the premerger is actually doing what is in the configuration and report into /var/log/syslog if a livesearch is missed or delayed (currently set to warn when a livesearch is . The script is designed to run from /root via cron - and will require root privileges to run as is uses tcpdump on lo:5555.

The header of the script will instruct how to setup the root crontab to get it to run properly - basically add the following line too root crontab using the crontab command:

life.py

This script is an improved version of the somewhat "hidden" script named lifegrp.py which is/was part of the premerger service installation. The script doesn't need privileges to run and will perform a lookup on a given life_id or lifegrp to list information about the owner, query, repo selection, time range etc of the livesearch - this can be used to identify potential optimisations and direct the information towards the owner of the livesearch.

The scripts will be updated occasionally so do drop by when in the neighborhood ;-)

Questions are always welcome!

Enjoy!

Peter Melsen

Bhabesh Raj,  our Associate Security Analytics Engineer at LogPoint recently shared his thoughts on how to prepare to fend off  PrintNightmare with LogPoint.  Do you have any experience with PrinterNightmare? If so, share your  advise or input for your fellow LogPoint users :)

The blog post can be reached on the link below:

Preparing your defenses against the terror of PrintNightmare (5 minutes read)

A dashboard can go blank for various reasons,  check the following things to correct the blank dashboard before contacting LogPoint Support.

1. Hit the query that is being used by the dashboard in the search bar and check if it has any matching logs.
2. If you do not find any matching logs, then there is a chance that correct normalization packages are not in place for the corresponding normalization policy.
3. If logs are shown and the dashboard is still blank,  check if the source devices are configured to the correct time zone from the device configuration settings.
4. If  the widget is still empty even after correcting the timezone of a device,  try to add a “dummy dashboard“ or try to edit that specific dashboard. It will trigger the dashboard engine to regenerate the config and also helps to check if there is a problem with a single dashboard or in dashboards in general.
5. If  the widget is still blank, create a support ticket with the below information:
   • The query being used in the dashboard.
   • Timerange of the query.
   • Logs inside /opt/immune/var/log/service/premerger/current. If query used is correlation query then also include logs from file /opt/immune/var/log/service/analyzer/current.

1. Decide the mode you want to configure zpool in (normal, mirror, raidz) and prepare the disks accordingly.

2. Set all the disks label to GPT:

parted /dev/sdX   # get disk-id or disk-path from /dev/disk/by-id or /dev/disk/by-path

mklabel gpt

3. Create a new zpool with following command:

zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 data_pool /dev/disk/by-id/<disk-id>

if in raidz mode, at least 3 disks are needed:

zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 data_pool   raidz /dev/disk/by-id/<disk1-id> /dev/disk/by-id/<disk2-id> /dev/disk/by-id/<disk3-id>

in mirror mode, at least two disks are needed.

zpool create -o ashift=12 -O atime=off -O canmount=off -O compression=lz4 data_pool mirror /dev/disk/by-id/<disk1-id> /dev/disk/by-id/<disk2-id>

4. Mount the pool to appropriate mountpoint:

zfs create -o mountpoint=<mount_point>  <poolName>/<filesystemName>

eg.  zfs create -o mountpoint=/data/store  Tank1/primary

5. Add the mount point to repo path. From li-admin, run:

repodir add <mount_point>

eg. repodir add /data/store

When creating a new normalization request, please include the following information to accelerate the norm package creation process:

• Name and version of the log source
• Logging documentation for the version of the log source (can be obtained from the device vendor)
• Sample logs from the device (The sample logs can be anonymised before being provided on the request)

Notes: You can use the following query to extract the un-normalized sample logs from the UI of LogPoint

device_ip='xx.xx.xx.xx' -norm_id=* | norm <ALL:.*> | fields ALL

Before running any upgrades, it is always recommended that you go through the release notes, and check up on any important notices that have been included. It is also important that you verify the upgrade pak file's SHA256 present on the release notes to that on LogPoint.

If you are using a virtual machine to run LogPoint, it is highly advised that you take a snapshot before running the upgrade. This is to make sure that you have a point to rollback if the upgrade runs into any issues.

If the upgrade does fail, please reach out to the LogPoint support team with the output of the following commands provided on a new HelpCenter request:

• tail -n 20 /opt/immune/var/patches/log/*6.10.0*.log (Replace the version of the LogPoint upgrade which failed)
• tail -n 20 /var/log/*6.10.0*.log (Replace the version of the LogPoint upgrade which failed)

Note: Please do not attempt a server reboot during the upgrade process, unless discussed prior with the support team.

There may be numerous reasons for the LPA configuration to not be pushed from LogPoint to the agents. The following are some of the things that you can easily check before contacting LogPoint Support.

1. Check if the Windows machine is actually running or not. This might seem trivial, but there are instances where the windows machines are not functioning in the first place.
2. Run the telnet command on the device for port 7994 and 7995. Since these are the ports via which the configurations are pushed, this might be an issue with the lpagents which are deployed on the log source. You can also check if  ports 7994 or 7995 are open on the device's end as well.
3. Verify that the configured device is not configured as NAT IP. If NAT IPs are being used, then the configuration will need to be copied manually to the correct directory; C:\Program Files (x86)\lpagent\conf\lpagent.conf
4. If the configuration is still not being pushed, create a support ticket with the below information:
   • LPAgent log files present on C:\Program Files (x86)\lpagent\data\lpagent.log
   • Logs present on /opt/immune/var/log/service/lpa_manager/current

If the browser reads Error 500, the following can be checked.

1. If the LogPoint server has just been booted up, the system will take some time to get the services up and running. Hence, simply waiting for 2-3 minutes should work in most cases.
2. Check the disk space utilization on the CLI using the command df -hT . If the /opt partition has less then 3.5GB, or the /opt/immune/storage or the root partition is at 100% utilization, this behavior may be seen. If this is the case, then either adding additional space or removing some of the files will be required. For this, the sudo access might be required, if so, then please create a support ticket with LogPoint Support.
3. If you are still not getting the UI to respond, then please create a new support ticket with the below information
   • Output of the df -hT command
   • Logs inside /opt/immune/var/log/service/webserver/current and /opt/immune/var/log/service/mongodb/current

Please carry out the following steps if the UEBA dashboard is blank.

1) Check the Settings>System>System Settings under the General tab. If the Apply Timerange On is selected to col_ts , change it to log_ts and check the dashboard again.

2) Go to the page Settings>Configuration>UEBA Board>Overview. Check the values of logs sent since last day, if all these values are zero, refer to <another community post:

3) Go to the search tab and see if any data is present in the uebaoutput repo for the last 24 hours. If no data is present, make a support request to LogPoint complaining about the unavailability of analytics result.

If you found out that the logs sent since last day values are all zero in Settings>Configuration>UEBA Board>Overview follow the given steps.

1) Confirm that the UEBA dockers are running properly. The following command should output a list of 3 dockers running:

 docker ps

2) Run tcpdump in the port 5532 of data server. If data are coming in, check if your firewall is blocking the connection to UEBA cluster. The IPs of UEBA cluster are provided during UEBA onboarding.

tcpdump -Aqs0 -i any port 5532

3) If there is no data in 5532, run tcpdump on the port 5530 and see if data is coming into the port. If data is flowing in, there is a possible a problem with licensing. Please check the Settings>Configuration>UEBA Board>Entity Selection and check from the search tab that the logs belonging to the selected Entities are actually present. Then, please make correct entity selection.

4) If data are not flowing into 5530, there is problem with log collection. Please contact LogPoint support.

Fields available in simple search

The search queries with field=value, word(s), phrases or logical combination of such without any use of "|" commands are defined as simple search. Each result row in the result of a simple search query can be taken as a set of field=value pairs and the actual logs. All the field=value pairs that are generated during normalization can be accessed. In addition, the following fields are added by default; log_ts, col_ts, device_name,device_ip,col_type.

example: Users failing to login
label=User label=Login label=Fail

Fields available in aggregation query results

There may be use-cases where a simple search doesn't suffice and there‘s a need to make aggregations based on one or more fields.

Example:

label=User label=Login label=Fail | chart count() as FailCount by target_user

The result of this query is a set of target_user values and the corresponding count, represented as FailCount. Only these two variables are available at this stage. If you want to make a search of results with fails more than hundred then you can do the following.

label=User label=Login label=Fail | chart count() as FailCount by target_user | search FailCount>100

Let's take a non-working example.

label=Allowed label=Connection | chart count() as CNT by source_address,destination_address | search destination_port <1024

The result of the query "label=Allowed label=Connection" will be a set of logs and field-value pairs and destination_port will also be contained on each result row. However, when "| chart count() as CNT by source_address,destination_address" is done we are basically reducing the result set in such a way that each row consists of three field-value pairs viz source_address,destination_address,CNT. This can also be seen as a table with 3 columns. If you now need to search for all connections that went to ports below 1024 you can't do "| search destination_port <1024" because the field destination_port is not available at this stage.

If you need to make search on the destination_port then you can do the following.

label=Allowed label=Connection | chart count() as CNT by source_address,destination_address,destination_port | search destination_port <1024 | chart sum(CNT) by source_address,destination_address

To make the destination_port available for the search segment "| search destination_port < 1024" we explicitly carried that in the first aggregation. We then filtered out all rows corresponding to destination_port < 1024 with the query "| search destination_port <1024". Finally we used another aggregation to get the sum of counts for each source_address,destination_address pair.

Alternatively (and optimally), this query could have been written as given below. The above was taken for sake of clarifying the usage.

label=Allow label=Connection destination_port<1024 | chart count() as CNT by source_address,destination_address

Fields available in join queries

The result of join queries are again rows of fields=value pairs. The raw logs are not included. If you are joining two result sets s1 and s2 then all the fields that are part of the result set can be referenced appending with s1. or s2. as applicable.

For example

[source_address=*] as s1 join [device_ip=*] as s2 on s1.source_address=s2.device_ip

In this case, all the fields that are part of the result set source_address=* can be accessed as s1.field_name. For example if the result set have a field called destination_port you will be able to access that as s1.destination_port. Likewise, all the fields in the result set of the second can be accessed as s2.field_name. You can rename the fields and do further aggregations as necessary.

Fields available in stream queries

[5 label=User label=Login label=Fail having same target_user within 5 minutes]

In this case, only the log_ts and the target_user is available for further aggregations and analysis.

Fields available in alert results

An alert is fired when a set of logs matches some pre-specified condition. When an alert is fired, the following variables are exposed to the notification interface.

alert_name, description,risk_level,detection_timestamp,search_link,rows_count,rows.

The rows_count contains the count of the the events particiapting in firing the alert. The variable rows is the set of results of the search query firing the alert. For notification each item in the result set can be accessed by iterating on this list using jinja templating.

For example if the query is as below

device_name=some_device| chart count() as CNT by source_address

You can use the following jinja template to access the values in the returned list.

{% for row in rows %}

{{row.source_address}}:{{row.CNT}}

{% endfor %}

The fields available in the generated alert for  stream queries are slightly different. For example take the below query.
[5 label=User label=Login label=Fail having same target_user within 5 minutes]

The jinja templating can also be used to access the fields in the participating events. For details please refer

Jinja Template made easy for Alert Notification

Issue and Observation

The message "AQL: graph 'mssp_db' not found (while parsing)" pops up while using the Director Console UI.

The error indicates that the synchronization between arango and zookeeper is in progress and sometimes due to stale entry of any fabric enabled logpoints, the sync can get halted.

Solution

Log files at /opt/commander/bigstore/var/log/directorconsole/  can provide useful information

Check if the fabric enabled Logpoint is present in zookeeper

ssh into fabric server

/opt/commander/installed/zookeeper/bin/zkCli.shaddauth digest zkacl:Welcome@123ls data/<POOL_IDENTIFIER>/addauth digest commander:<enter password> ls  /license/<LP identifier>get /license/<LP identifier>delete any stale or duplicate logpoint identifiers if present withrmr /<node id>

Trigger data fetch fron arangodb GUI

ssh into Director API

Edit /etc/ssh/sshd_config to enable ssh local port forwarding ,

AllowTcpForwarding yes

ssh -L 8529:127.0.0.1:8529 support@<logpoint ip>

Open chrome/firefox browser and enter url http://127.0.0.1:8529

Login as root/changeitSelect: mssp_db

select TimeStamp collection from

Collection > TimeStamp/progress

Change the value to “in_progress” and save the changes

ssh into the Director Console machine and restart the dc-front and dc-back containers with commands below:-

docker stop dc-frontdocker stop dc-backdocker start dc-frontdocker start dc-back

During the installation of the DC/API patch update the containers might not be in running state due to unknown reasons, you can check the status of containers by running the command below:

docker ps # output from DC v1.7.0Below is output of docker ps on DC v1.8.0

Observation

In an ideal state, all three containers -dc-front, dc-back,arangodb are in a running and in healthy state (in DC v 1.8.0 there are 4 containers where tomcat is added). However in some patch upgrade cases the containers may be missing.

Solution

To check if the container is present but not running:

docker ps -a  

If the container is shown, start the container with the following command:

docker start <container ID>

Now, continue with:

installcom-patch /path/to/<patch file>

If the container is not present run the command below:

cd /opt/commander/installed/system/cmdr-admin/root-actions

docker-compose -p dc up -d

The above command uses docker-compose.yml file to bring the containers up.