Introduction – Audit Logs
Business Group Aware logging to identify access violation.
Different user’s activity such as type of action performed for application over an environment is captured here.
Payload can also be downloaded from here for the specific action performed by the user.
Its use typically depends on organizational and legal requirements.
It uses the Audit Logging Query API.
It Can be viewed for all the users within business group.
These logs can be downloaded.
They are retained for 6 years.


Introduction – System Logs
System Logs are specific to Mule Runtime.
Log messages about the Mule runtime lifecycle (startup and shutdown).
Includes System.out log messages only in case of customer hosted run-time planes.
In case of on-prem, access mule_ee.log for system logs.

For CloudHub, go to Runtime Manager à Application à Logs à System Logs.
It contains status messages about mule applications.

Introduction – Application Logs
It Specific to the Mule application. Types of application logging present in Mule: DEBUG, WARN, ERROR, INFO (Default).
Configured by log4j2.xml typically packaged in Mule application. It provides all the application logs as defined either in application’s log4j2 and associated log4j2.xml appenders or cloudhub’s log4j2.
Logs are distributed across number of workers allotted to application and can be downloaded.
CloudHub truncates log messages up to 100 MB per Mule application and per worker (at most 30 days)
All log messages are generated inside the Mule application.
Runtime Manager can override the log level for a deployed Mule application without redeployment.
Just need to add package name to runtime manager à application à settings à logging tab. Set the type of error you need for that package.
This package name corresponds to the connector or module package for which you would want to get logs.

Uses of log4j2.xml
Log4j2.xml contains all the information which helps to generate application logs.
The correlation id is logged by the default log4j2.xml as event.
Logs can be configured using various appender.
It is Used for connectors and modules to record debug level information in the logs.

Log4j2.xml can be customized as desired, this file is by default ignored by CloudHub.
By default, during application deployment, CloudHub uses its own log4j2.xml and replaces log4j2.xml of the application.
During Deployment, select “Disable Cloudhub Logs” to use Application’s log4j2.xml.

Enable Verbose Logging
Verbose logging can be used for different modules and connectors.
Because of the high impact to application performance, limit verbose logging to testing and troubleshooting operations.
Add entry to log4j2.xml file indicating package names of the connectors or modules you want to log (from package explorer) and log level = DEBUG.

Synchronous and Asynchronous Logging
Synchronous Logging:
The execution of the thread that is processing your message is interrupted to wait for the log message to be fully output before it can continue.
Performance degrades because of synchronous logging.
Used when the log is used as an audit trail or when logging ERROR/CRITICAL messages
Asynchronous Logging (Default):
The logging operation occurs in a separate thread, so the actual processing of your message won’t be delayed to wait for the logging to complete
Substantial improvement in throughput and latency of message processing.
Log may be lost in case of system crash.
References:
https://docs.mulesoft.com/mule-runtime/4.3/logging-in-mule
https://docs.mulesoft.com/mule-runtime/4.3/logging-and-debugging
https://docs.mulesoft.com/mule-runtime/4.3/enable-verbose-logging