Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overhaul of the Logging Infrastructure #1596

Draft
wants to merge 40 commits into
base: dev
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
0293115
Pulled ostream operators for IPAddress and MacAddress into the namesp…
Dimi1010 Sep 20, 2024
45115e3
Refactor of Logger.
Dimi1010 Sep 30, 2024
dfb98c5
Merge remote-tracking branch 'upstream/dev' into refactor/log-templates
Dimi1010 Sep 30, 2024
9062c48
Cleanup and fixes.
Dimi1010 Sep 30, 2024
1e856f0
Added the new Off log level to the string conversion.
Dimi1010 Sep 30, 2024
02618fc
Fixed wrong variable name.
Dimi1010 Sep 30, 2024
32ff8f9
Added documentation to log source.
Dimi1010 Sep 30, 2024
12492d2
Lint.
Dimi1010 Sep 30, 2024
f21f85d
Fixed docstring for LogSource.
Dimi1010 Sep 30, 2024
0faf2e8
Fixed extra /
Dimi1010 Sep 30, 2024
ae72e81
Merge remote-tracking branch 'upstream/dev' into refactor/log-templates
Dimi1010 Oct 6, 2024
43d7b1c
Fixed explicit warning.
Dimi1010 Oct 6, 2024
87042f5
Merge remote-tracking branch 'upstream/dev' into refactor/log-templates
Dimi1010 Oct 14, 2024
be95ab8
Moved log functions inside logger.
Dimi1010 Oct 14, 2024
4b21a99
Revert "Moved log functions inside logger."
Dimi1010 Oct 14, 2024
00034e9
Merge remote-tracking branch 'upstream/dev' into refactor/log-templates
Dimi1010 Nov 5, 2024
b2055c0
Moved the log functions to the Logger class.
Dimi1010 Nov 5, 2024
cfc3d56
Merge branch 'dev' into refactor/log-templates
Dimi1010 Nov 14, 2024
ac2a383
Fixed typo in macro names.
Dimi1010 Nov 14, 2024
3776653
Changed value param to const-ref.
Dimi1010 Nov 14, 2024
76079a7
Added "venv" and "./out" to ignored directories by codespell.
Dimi1010 Nov 14, 2024
52b9007
Reverted to previous optimizations to keep executable binary size low.
Dimi1010 Nov 14, 2024
c71dd27
Fixed warnings about unreferenced local variables if the compile time…
Dimi1010 Nov 14, 2024
803c98a
Removed useless variable.
Dimi1010 Nov 14, 2024
9d44f76
Fixed friend class definition.
Dimi1010 Nov 14, 2024
af63a9b
Fixed variable assignment.
Dimi1010 Nov 14, 2024
02e717a
Added method useContextPooling to control if the logger should use co…
Dimi1010 Nov 16, 2024
3410231
Fixed more warnings about unreferenced local variables if the compile…
Dimi1010 Nov 16, 2024
379c074
Addressed warnings and documentation.
Dimi1010 Nov 16, 2024
bc82caf
Fixed include.
Dimi1010 Nov 16, 2024
69aa615
Fixed pointer dereference.
Dimi1010 Nov 20, 2024
17ec9a2
Fixed memory checker issues with logger.
Dimi1010 Nov 20, 2024
2029b4b
Lint
Dimi1010 Nov 20, 2024
c09a11a
Added mutex lock on the default log printer to support proper multi-t…
Dimi1010 Nov 20, 2024
f1b7cd6
Merge remote-tracking branch 'upstream/dev' into refactor/log-templates
Dimi1010 Dec 25, 2024
6b8c104
Fixed typos in documentation.
Dimi1010 Dec 25, 2024
e972e67
Changed level variable to private.
Dimi1010 Dec 25, 2024
85ae097
Changed LogPrinter definition to use the metaprogramming construct st…
Dimi1010 Dec 25, 2024
a1e7d56
Replaced C library includes with C++ equivalents.
Dimi1010 Dec 25, 2024
445197b
Updated documentation format.
Dimi1010 Dec 25, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 31 additions & 31 deletions Common++/header/IpAddress.h
Original file line number Diff line number Diff line change
Expand Up @@ -1082,40 +1082,40 @@ namespace pcpp
std::unique_ptr<IPv4Network> m_IPv4Network;
std::unique_ptr<IPv6Network> m_IPv6Network;
};
} // namespace pcpp

inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv4Address& ipv4Address)
{
os << ipv4Address.toString();
return os;
}
inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv4Address& ipv4Address)
{
os << ipv4Address.toString();
return os;
}

inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv6Address& ipv6Address)
{
os << ipv6Address.toString();
return os;
}
inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv6Address& ipv6Address)
{
os << ipv6Address.toString();
return os;
}

inline std::ostream& operator<<(std::ostream& os, const pcpp::IPAddress& ipAddress)
{
os << ipAddress.toString();
return os;
}
inline std::ostream& operator<<(std::ostream& os, const pcpp::IPAddress& ipAddress)
{
os << ipAddress.toString();
return os;
}

inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv4Network& network)
{
os << network.toString();
return os;
}
inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv4Network& network)
{
os << network.toString();
return os;
}

inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv6Network& network)
{
os << network.toString();
return os;
}
inline std::ostream& operator<<(std::ostream& os, const pcpp::IPv6Network& network)
{
os << network.toString();
return os;
}

inline std::ostream& operator<<(std::ostream& os, const pcpp::IPNetwork& network)
{
os << network.toString();
return os;
}
inline std::ostream& operator<<(std::ostream& os, const pcpp::IPNetwork& network)
{
os << network.toString();
return os;
}
} // namespace pcpp
199 changes: 148 additions & 51 deletions Common++/header/Logger.h
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
#include <sstream>
#include <iomanip>
#include <stdint.h>
#include "DeprecationUtils.h"

#ifndef LOG_MODULE
# define LOG_MODULE UndefinedLogModule
Expand All @@ -17,29 +18,6 @@
# define PCAPPP_FILENAME __FILE__
#endif

#define PCPP_LOG(level, message) \
do \
{ \
std::ostringstream* sstream = pcpp::Logger::getInstance().internalCreateLogStream(); \
(*sstream) << message; \
pcpp::Logger::getInstance().internalPrintLogMessage(sstream, level, PCAPPP_FILENAME, __FUNCTION__, __LINE__); \
} while (0)

#define PCPP_LOG_DEBUG(message) \
do \
{ \
if (pcpp::Logger::getInstance().logsEnabled() && pcpp::Logger::getInstance().isDebugEnabled(LOG_MODULE)) \
{ \
PCPP_LOG(pcpp::Logger::Debug, message); \
} \
} while (0)

#define PCPP_LOG_ERROR(message) \
do \
{ \
PCPP_LOG(pcpp::Logger::Error, message); \
} while (0)

/// @file

/**
Expand Down Expand Up @@ -79,6 +57,7 @@ namespace pcpp
PacketLogModuleGreLayer, ///< GreLayer module (Packet++)
PacketLogModuleSSLLayer, ///< SSLLayer module (Packet++)
PacketLogModuleSllLayer, ///< SllLayer module (Packet++)
PacketLogModuleSll2Layer, ///< Sll2Layer module (Packet++)
PacketLogModuleNflogLayer, ///< NflogLayer module (Packet++)
PacketLogModuleDhcpLayer, ///< DhcpLayer module (Packet++)
PacketLogModuleDhcpV6Layer, ///< DhcpV6Layer module (Packet++)
Expand Down Expand Up @@ -113,10 +92,68 @@ namespace pcpp
PcapLogModuleDpdkDevice, ///< DpdkDevice module (Pcap++)
PcapLogModuleKniDevice, ///< KniDevice module (Pcap++)
PcapLogModuleXdpDevice, ///< XdpDevice module (Pcap++)
NetworkUtils, ///< NetworkUtils module (Pcap++)
PcapLogModuleNetworkUtils, ///< Network Utils module (Pcap++)
NumOfLogModules
};

/**
* @struct LogSource
* Represents the source of a log message.
* Contains information about the source file, function, line number, and the log module.
*/
struct LogSource
{
/**
* Default constructor for LogSource.
*/
constexpr LogSource() = default;

/**
* Constructor for LogSource with only the log module.
* @param logModule The log module.
*/
explicit constexpr LogSource(LogModule logModule) : logModule(logModule)
{}

/**
* Constructor for LogSource with all parameters.
* @param logModule The log module.
* @param file The source file.
* @param function The source function.
* @param line The line number.
*/
constexpr LogSource(LogModule logModule, const char* file, const char* function, int line)
: file(file), function(function), line(line), logModule(logModule)
{}

const char* file = nullptr; /**< The source file. */
const char* function = nullptr; /**< The source function. */
int line = 0; /**< The line number. */
LogModule logModule = UndefinedLogModule; /**< The log module. */
};

/**
* An enum representing the log level. Currently 3 log levels are supported: Error, Info and Debug. Info is the
* default log level
*/
enum class LogLevel
{
Off, ///< No log messages are emitted.
Error, ///< Error level logs are emitted.
Info, ///< Info level logs and above are emitted.
Debug ///< Debug level logs and above are emitted.
};

inline std::ostream& operator<<(std::ostream& s, LogLevel v)
{
return s << static_cast<std::underlying_type<LogLevel>::type>(v);
}

// Forward declaration
template <class T> void log(LogSource source, LogLevel level, T const& message);
template <> void log(LogSource source, LogLevel level, std::string const& message);
template <> void log(LogSource source, LogLevel level, const char* const& message);

/**
* @class Logger
* PcapPlusPlus logger manager.
Expand All @@ -139,16 +176,14 @@ namespace pcpp
class Logger
{
public:
/**
* An enum representing the log level. Currently 3 log levels are supported: Error, Info and Debug. Info is the
* default log level
*/
enum LogLevel
{
Error, ///< Error log level
Info, ///< Info log level
Debug ///< Debug log level
};
// Deprecated, Use the LogLevel in the pcpp namespace instead.
using LogLevel = pcpp::LogLevel;
PCPP_DEPRECATED("Use the LogLevel in the pcpp namespace instead.")
static const LogLevel Error = LogLevel::Error;
PCPP_DEPRECATED("Use the LogLevel in the pcpp namespace instead.")
static const LogLevel Info = LogLevel::Info;
PCPP_DEPRECATED("Use the LogLevel in the pcpp namespace instead.")
static const LogLevel Debug = LogLevel::Debug;

/**
* @typedef LogPrinter
Expand Down Expand Up @@ -196,7 +231,18 @@ namespace pcpp
*/
bool isDebugEnabled(LogModule module) const
{
return m_LogModulesArray[module] == Debug;
return m_LogModulesArray[module] == LogLevel::Debug;
}

/**
* @brief Check whether a log level should be emitted by the logger.
* @param level The level of the log message.
* @param module PcapPlusPlus module
* @return True if the message should be emitted. False otherwise.
*/
bool shouldLog(LogLevel level, LogModule module) const
{
return level != LogLevel::Off && m_LogModulesArray[module] >= level;
}

/**
Expand Down Expand Up @@ -259,20 +305,6 @@ namespace pcpp
return m_LogsEnabled;
}

template <class T> Logger& operator<<(const T& msg)
{
(*m_LogStream) << msg;
return *this;
}

std::ostringstream* internalCreateLogStream();

/**
* An internal method to print log messages. Shouldn't be used externally.
*/
void internalPrintLogMessage(std::ostringstream* logStream, Logger::LogLevel logLevel, const char* file,
const char* method, int line);

/**
* Get access to Logger singleton
* @todo: make this singleton thread-safe/
Expand All @@ -284,17 +316,82 @@ namespace pcpp
return instance;
}

template <class T> friend void pcpp::log(LogSource source, LogLevel level, T const& message);

private:
bool m_LogsEnabled;
Logger::LogLevel m_LogModulesArray[NumOfLogModules];
LogLevel m_LogModulesArray[NumOfLogModules];
LogPrinter m_LogPrinter;
std::string m_LastError;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seladb Do we absolutely need to store the last error? Because the way it works, if the logger is made to be used by more than one thread the value is completely useless as it can be changed at any time by any thread.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, however we currently expose the getLastError() method which might be used by some users so we can't just remove it...

Copy link
Collaborator Author

@Dimi1010 Dimi1010 Jan 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so we are left with the options:

  • keep the current functionality and only fix the potential tearing issues. All threads save to the singular last error variable.
  • change the last error to thread local scope, which would essentially only keep the main thread errors. Other errors called from background threads (like capture threads) will not be able to be captured by the user via the getLastError method as it eould be a different thread.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can leave the current functionality for now. It's not perfect but probably good enough and no one complained so far 🙂

std::ostringstream* m_LogStream;

// private c'tor - this class is a singleton
Logger();

void printLogMessage(LogSource source, LogLevel logLevel, std::string const& message);
static void defaultLogPrinter(LogLevel logLevel, const std::string& logMessage, const std::string& file,
const std::string& method, const int line);
};

template <class T> inline void log(LogSource source, LogLevel level, T const& message)
{
auto& logger = Logger::getInstance();
if (logger.shouldLog(level, source.logModule))
{
std::ostringstream sstream;
sstream << message;
logger.printLogMessage(source, level, sstream.str());
}
};

// Specialization for string to skip the stringstream
template <> inline void log(LogSource source, LogLevel level, std::string const& message)
{
auto& logger = Logger::getInstance();
if (logger.shouldLog(level, source.logModule))
{
logger.printLogMessage(source, level, message);
}
};

// Specialization for const char* to skip the stringstream
template <> inline void log(LogSource source, LogLevel level, const char* const& message)
{
auto& logger = Logger::getInstance();
if (logger.shouldLog(level, source.logModule))
{
logger.printLogMessage(source, level, message);
}
};

template <class T> inline void logError(LogSource source, T const& message)
{
log(source, LogLevel::Error, message);
};

template <class T> inline void logInfo(LogSource source, T const& message)
{
log(source, LogLevel::Info, message);
};

template <class T> inline void logDebug(LogSource source, T const& message)
{
log(source, LogLevel::Debug, message);
};
} // namespace pcpp

#define PCPP_LOG(level, message) \
do \
{ \
if (pcpp::Logger::getInstance().shouldLog(level, LOG_MODULE)) \
{ \
std::ostringstream sstream; \
sstream << message; \
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please check if this change affects the PcapPlusPlus binary sizes?

I made a change in this PR a couple of years ago to significantly reduce the binary sizes: #748

Copy link
Collaborator Author

@Dimi1010 Dimi1010 Oct 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a breakdown on the install directory for the x64 release w/npcap.

Folder1 is the new size. Folder2 is the old size.

File                                                     Folder1Size Folder2Size AbsoluteDelta RelativeDelta
----                                                     ----------- ----------- ------------- -------------
bin\Arping.exe                                                714240      639488         74752 10.47%
bin\ArpSpoofing.exe                                           703488      631296         72192 10.26%
bin\benchmark.exe                                             599552      562688         36864 6.15%
bin\DisplayLiveDevices.exe                                     30208       30208             0 0.00%
bin\DNSResolver.exe                                           748032      664064         83968 11.23%
bin\DnsSpoofing.exe                                           747520      671232         76288 10.21%
bin\HttpAnalyzer.exe                                          832512      743936         88576 10.64%
bin\IcmpFileTransfer-catcher.exe                              730624      658432         72192 9.88%
bin\IcmpFileTransfer-pitcher.exe                              742400      666112         76288 10.28%
bin\IPDefragUtil.exe                                          711680      655360         56320 7.91%
bin\IPFragUtil.exe                                            693248      637952         55296 7.98%
bin\PcapPrinter.exe                                           658432      612352         46080 7.00%
bin\PcapSearch.exe                                            661504      618496         43008 6.50%
bin\PcapSplitter.exe                                          720384      667648         52736 7.32%
bin\SSLAnalyzer.exe                                           822272      733184         89088 10.83%
bin\TcpReassembly.exe                                         799232      710144         89088 11.15%
bin\TLSFingerprinting.exe                                     796672      712704         83968 10.54%
lib\Common++.lib                                             5045102     4887832        157270 3.12%
lib\Packet++.lib                                            19849740    18454316       1395424 7.03%
lib\Pcap++.lib                                               3620658     2896688        723970 20.00%

Full breakdown: npcap-x64-release.txt

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you test it with MSVC or MinGW? Also, did you have a chance to test on Linux also?

I don't remember in which platform the impact was the highest, but I think it was Linux 🤔

Copy link
Collaborator Author

@Dimi1010 Dimi1010 Oct 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Msvc, I dont think I have mingw setup. I should be able to test the sizes on Unix via WSL.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Dimi1010 I see you're still working on it. Please let me know when the changes are done and when you want me to review the PR again.
It'd be good to make sure binary sizes don't increase. I'd test at least on Windows and Linux

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seladb Ok, ended up keeping the optimization.

As bc82caf: Release builds ended up being:

  • Windows
    • Static Libraries
      • Common++.lib - 4.7MB -> 5MB
      • Packet++.lib - 17.6MB -> 19.7MB
      • Pcap++.lib - 2.8MB -> 3.3MB
    • Examples
      • Same binary size or <=10KB increase.
  • Linux (WSL)
    • Static Libraries
      • libCommon++.a - 408KB -> 444KB
      • libPakcet++.a - 3.1MB -> 3.3MB
      • libPcap++.a - 676KB -> 748KB
    • Examples
      • On average ~0.1MB increase on each binary.

pcpp::log(pcpp::LogSource(LOG_MODULE, PCAPPP_FILENAME, __FUNCTION__, __LINE__), level, sstream.str()); \
} \
} while (0)

#define PCPP_LOG_DEBUG(message) PCPP_LOG(pcpp::LogLevel::Debug, message)

#define PCPP_LOG_INFO(message) PCPP_LOG(pcpp::LogLevel::Info, message)

#define PCPP_LOG_ERROR(message) PCPP_LOG(pcpp::LogLevel::Error, message)
12 changes: 6 additions & 6 deletions Common++/header/MacAddress.h
Original file line number Diff line number Diff line change
Expand Up @@ -173,10 +173,10 @@ namespace pcpp
private:
uint8_t m_Address[6] = { 0 };
};
} // namespace pcpp

inline std::ostream& operator<<(std::ostream& os, const pcpp::MacAddress& macAddress)
{
os << macAddress.toString();
return os;
}
inline std::ostream& operator<<(std::ostream& os, const pcpp::MacAddress& macAddress)
{
os << macAddress.toString();
return os;
}
} // namespace pcpp
Loading
Loading