plik

www.redhat.com

USING aN oPEN SoURCE

fRaMEWoRK to CatCH tHE baD GUY

Built-in forensics, incident

response, and security with

red hat enterprise linux 6

red hat federal solutions architect team
By: norman mark st. laurent, senior solutions architect
editing and technical guidance: shawn wells, technical director
steve gruBB, red hat security lead

Part 1: Operational Use of the Red Hat
Enterprise Linux 6 Audit Log Management
Infrastructure

1.1 Establishing Policies and Procedures
for Log Management

1.1.1 RHEL 6 Log Storage and Rotation

1.1.1.1 Non auditd Log Files in /var/log

1.1.1.2 auditd Log Files in the Default /var/
log/audit Directory

1.1.2 RHEL 6 Remote Host Storage

1.1.2.1 Log Management with the /etc/audit/
auditd.conf File

1.1.2.1.1 Encryption over the Wire With SSH
Port Forwarding

1.1.2.2 Log Management with the /etc/
audisp/audispd.conf File

1.1.2.2.1 Log Management with the /etc/
audit/audisp-remote.conf File

1.1.3SpecificRHEL6LogGeneration
Settings

1.1.3.1 Log Management with the /etc/audit/
audit.rules File

1.2 Red Hat Enterprise Linux 6 Log
Management Operational Process

1.2.1DefiningRolesandResponsibilities

1.2.2 RHEL 6 Forensics and Incident
Response Log Analysis

Part 2: Host-based Intrusion
Detection System

40 Bibliography

2 www.redhat.com

abStRaCt

Every security policy provides guidance and requirements for ensuring adequate protection of information
and data, as well as high level technical and administrative security requirements for a system in a given
environment.Traditionally,providingsecurityforasystemhasfocusedontheconfidentiality

of the informa-

tion on it; however protecting the data integrity

and system and data availability

are just as important. For

example, for processing United States intelligence information there are three attributes that require protec-

tion:confidentiality,integrity,andavailability.

InordertorunonUnitedStatesGovernmentSystems,RedHat

Enterprise Linux

6 has met the a stringent

setoftechnicalsecurityrequirementsforconfidentiality,integrity,andavailabilitytoallowconformance
tobecertifiedandaccredited

.RedHatEnterpriseLinuxhasreceivedCommonCriteriacertificationat

EnterpriseAssuranceLevel4(EAL4+)undertheControlledAccessProtectionProfile(CAPP),LabelSecurity
ProtectionProfile(LSPP),andtheRole-BasedAccessProtectionProfile(RBACPP).

Security for Red Hat Enterprise Linux 6 begins with a core feature known as SELinux. SELinux delivers
astrongandflexibleMandatoryAccessControl(MAC)frameworktoenforcerole-basedaccesscontrol
and multi-level security. Security-Enhanced Linux support has been woven into all parts of the platform,
including virtualization, to provide critical guest separation regardless of the guest operating system.
SuccessfulsecurityusesaDefense-in-Depthstrategy,soRHEL6includessystemfirewalls

, host base intru-

siontools,systempackageandfileintegrityverificationtools,andasdiscussedinPART 1 of this whitepaper,
audit capabilities for a complete security architecture that covers deployment models ranging from Internet-
facing servers to trusted computing.

Common Criteria security event auditing requirements are covered in both the CAPP and LSPP protection
profiles.CAPPwasderivedfromtheOrangeBook

C2criteriaanddefinesaudittoprovidecomprehensive

logging of security events that are reliable and robust. LSPP extends audit requiring “enhanced security
event auditing” to include Mandatory Access Control (MAC) labeling and decision information. LSPP was
derived from the Orange Book B1 criteria. table 1: capp requirements provides a detailed description of the

CAPP requirement for audit. Audit must be non-bypassable, and the right to add records to the audit trail
mustbecontrolled.Therequirementsalsonotethatbothsetting/viewingtheauditconfigurationmustbe
controlled and that audit review must be controlled and assignable. It must have the ability to fail-stop the
system. The Linux syslog

facility has none of these properties.

1 Confidentialityisdefinedasensuringthatinformationisaccessibleonlytothoseauthorizedtohaveaccess.

2 IntegrityisdefinedasqualityofanITsystemreflectingthelogicalcorrectnessandreliabilityoftheoperatingsystem;thelogical

completeness of the hardware and software implementing the protection mechanisms; the consistency of the data structures and
occurrence of the data stored.

3 Availabilityisdefinedastimely,reliableaccesstodataandinformationservicesforauthorizedusers.

4 RHEL has passed the Common Criteria Process 13 times on four different hardware platforms.

5 RHEL 6 also includes Ethernet bridge frame table administration ( ebtables).This application program is used to set up and maintain the

table of rules inside the kernel that inspects Ethernet frames. It works just like the iptables application, which inspects the IP protocol;
ebtables inspects the Ethernet protocol.

6 Trusted Computer System Evaluation Criteria (TCSEC), referred to as the Orange Book, is a DOD standard that sets basic requirements for

assessing the effectiveness of computer security controls built into a computer system.

7 RHEL 6 uses rsyslogd, which is a reliable and extended syslogd. It is a system utility providing support for message logging. Local and

remote logging is supported, but it is not as granular as auditd and does not meet the strict requirements of CAPP and LSPP.

Using an open source framework to catch the bad guy

www.redhat.com 3

TheFutureCommonCriteriaProtectionProfile:the Operating System perspective has changed from single

isolated systems to more complex distributed and networked environments (e.g., virtualization and cloud),
thusrenderingseveraloftheoriginalprotectionprofiles,includingLSPP,RBAC,andCAPP,lessrobuston
therequirements.OnthehorizonisasecondgenerationcertifiedOperatingSystemProtectionProfile

(OSPP). Red Hat is currently meeting the standards and requirements of OSPP from the networked systems
approach and will meet the functional and assurance requirements that are applicable. In addition, applica-

tions executing on operating systems depend upon a secure platform. The security assurance provided by
many modern operating systems has been raised over the last decade with EAL4 being the norm for this
technology and Red Hat raising the bar higher.

taBle 1: capp requirements

aUDIt toolS

DEfINItIoN

Audit data generation

The Target of Evaluation Security Functions (TSF )shall be able to generate
an audit record of the auditable events listed in column “Event” of Table
(Auditable Events). This includes all auditable events for the basic level of
audit, except FIA_UID.1’s user identity during failures.

Audit data generation

The TSF shall record within each audit record at least the following informa-
tion: (a) date and time of the event, type of the event, subject identity, and the
outcome(successorfailure)oftheevent;(b)additionalinformationspecified
in Table 1 .

User identity association

The TSF shall be able to associate each auditable event with the identity of the
user that caused the event.

Audit review

The TSF shall provide authorized administrators with the capability to read the
audit information from the audit records.

Audit review

The TSF shall provide authorized administrators with the capability to read all
audit information from the audit records.

Restricted audit review

The TSF shall prohibit all users read access to the audit records, except those
users that have been granted explicit read-access.

Prevention of audit data loss

ThiscomponentspecifiesthebehavioroftheTargetofEvaluation(TOE).If
the audit trail is full: either audit records are ignored, or the TOE is frozen such
that no auditable events can take place. The requirement also states that no
matterhowtherequirementisinstantiated,theauthorizeduserwithspecific
rights to this effect can continue to generate auditable events (actions).

Federal security policies also mention that there must be an in-depth strategy that provides appropriate
degrees of protection to all computing environments, hosts, and applications. Information systems should be
monitored in order to detect, isolate, and react to intrusions, disruption of services, or other incidents that
threaten the security. Requirements and recommendations for audit should be created in alignment with the
security implications as well as the regulations and laws to which the organization are subject. A number of
laws, policies, and regulations compel organizations to store and review audit data

Security policy requirements also state that there be a way to collect and retain audit data to support foren-
sics and incident response relating to misuse, penetration, reconstruction, or other investigations. During a
forensics investigation, law enforcement and analysts will need to rely on audit logs as a source of evidence.

Along with this, there must also be proof that a malicious person has not altered those logs and that the

8 Red Hat Enterprise Linux includes the openscap-utils package. This package is the Security Content Automation Protocol (SCAP)

toolkit based on the NSA/NIST OpenSCAP library (to include the Open Vulnerability and Assessment Language (oval), the eXtensible
ConfigurationChecklistDescriptionFormat(xccdf)theCommonePlatformEnumeration(CPE),andtheCommonVulnerabilityScoring
System (CVSS). For more information on Open SCAP visit: http://www.open-scap.org/doc/

9 LawsandRegulations:Commercial-FISMA,HIPAA,GLBA,SOX,PCIDSS//Government-CNSSDirectiveNo.502,DoDDirective8500

Series, NSD 32, DCID 6/3, DOD 5200.

Using an open source framework to catch the bad guy

4 www.redhat.com

logs are creditable. Logs produced by a computer are not admissible as evidence unless it can be shown that
there is no reasonable ground for believing them to be inaccurate, and the computer was operating properly
duringthecollectionofdata.Thelogfilescannotbetamperedwith,ortheyarenotadmissibleasevidence.

This two-part whitepaper series covers the critical areas of information protection for Red Hat Enterprise

Linux 6:

• Part1:UsingtheRedHatEnterpriseLinuxAuditSubSystems for forensics and incident response to

meet security requirement objectives and goals. This section of the white paper closely follows and maps
NIST Special Publication 800-92 GuidetoComputerSecurityLogManagement written by Karen Kent
andMurugiahSouppayawithRHEL6auditspecifics.

• Part2:IntegritycheckingwithRedHatEnterpriseLinux6, which involves calculating a message

digestforeachfileandstoringthemessagedigestsecurelytoensurethatchangestoarchivedlogsare
detected.Amessagedigest(alsocalledadigitalsignature)uniquelyidentifiesdataandhastheproperty

that changing a single bit in the data causes a completely different message digest to be generated.

aboUt tHE aUtHoRS

norman mark st. laurent, principal author

Norman Mark St. Laurent is a Senior Solutions Architect on the Red Hat Federal Team. Early in his 20 year

Computer Security/Forensics career, Mr. St. Laurent evaluated HPUX 10.10 as a Senior Computer Scientist for

the Trust Technology Assessment Program (TTAP) (Prelude to Common Criteria). Norman was also a Senior
Network Intrusion Analyst and Senior Information Systems Security Engineer (ISSE) for the NSA. In addition,
he was a Senior Computer Forensics Examiner (Unix/Linux Lead) for the FBI’s Computer Analysis Response

Team, having worked hundreds of computer and network forensics cases using Red Hat Linux as his primary

platform.

shawn wells, technical editor

Shawn Wells is the Technical Director for the Intelligence Community on the Red Hat Federal Team.

steve gruBB, technical editor

SteveGrubbleadsRedHat’ssecuritytechnologiesteam,whichworksonsecuritycertificationsandguidance
andmaintainsmanyofthesecuritytoolsthatyoufindonLinuxsystems,includingtheAuditSubSystem.He

hasworkedonLinuxsecurityforover10years,mostlyonflawdiscoveryandrepairformanyoftheimpor-
tant programs in use.

PaRt 1: oPERatIoNal USE of tHE RED Hat
ENtERPRISE lINUx 6 aUDIt
loG MaNaGEMENt INfRaStRUCtURE

The Red Hat Enterprise Linux 6 AuditingSubSystem provides kernel-resident logging of system calls and

user space tools to collect and view the logs, allowing for a means to provide both detailed and granular
forensics investigation as well as incident response. RHEL 6 allows for the capability to monitor real-time
occurrences of, or accumulation of, auditable events that may indicate an imminent violation of the security
policy.

Using an open source framework to catch the bad guy

www.redhat.com 5

In fact, the RHEL 6 AuditingSubSystemisconfigurabletoallowcontroloverwhatspecificinformationis
written to the logs. This information is useful in debugging security-related issues. The auditd daemon is
also used to log Security-Enhanced Linux (SELinux)

events. SELinux represents the culmination of nearly

40yearsofoperatingsystemsecurityresearchandprovidesapowerful,flexible,mandatoryaccesscontrol

mechanism to RHEL 6. SELinux generates audit messages at system initialization, policy load, and when
Boolean state are changed. These SELinux logs and the log management facility of auditd allow for security
relevanteventstobesecure,reliable,finegrained,andconfigurable,withavarietyofusesincluding:

• postmortem analysis,

• intrusion detection, and

• live system monitoring and debugging.

RHEL 6 audit logs are most useful for identifying or investigating suspicious activity involving a particular
host.Theauditlogscanbeconsultedtogetmoreinformationonaspecificactivityandallowaneventtobe
investigated at a more granular level. Built-in audit utilities such as aureport, ausearch, and aulast enable
an organization to view the audit log information in detail for analysis. These tools are very powerful and in
combination with shell scripting and the cron

facility can be very powerful as we’ll show later in this white-

paper. RHEL 6 also comes with the audit-viewer tool. The audit-viewertoolisaGraphicalUserInterface

(GUI)forviewingandsummarizingeventscollectedbytheauditsubsystem(seeFigure 1: The audit-

viewer GUI and Figure 2:The audit-viewer GUI with options shown).

The RHEL 6 Auditing Sub System also has the ability to monitor tty

logging that will log all tty sessions

(keystrokes) via the pam_tty_audit PAM

module. The pam_tty_audit PAM module is used to enable

or disable tty auditing. When tty auditing is enabled via this PAM module, it is inherited by all processes
startedbythatuser.Daemonsrestartedbyauserwillstillhavethisspecifickeystrokeauditingenabled.In

the session section of the /etc/pam.d/system-authfileyouneedtoaddthefollowinglineasshown
in Table 2: /etc/pam.d/system-auth pam_tty_audit addition to monitor all keystrokes the root
user has entered. It is recommended to use the disable=*asthefirstoptionsothatwillturnoffaudittty

input for all users except for the root user, whose we turn on with the enable=root option. Once set up, the

keystroke monitoring can be audited, and the data that was logged by the kernel can be seen by using the
aureport command (see Table 3:Reviewing keystroke captures with aureport). In the aureport

command we added the -ts today option to print out all keystrokes captured for the current day.

Table 2: /etc/pam.d/system-auth pam_tty_audit addition

/etc/pam.d/system-auth pam_tty_audit addition

Session required pam_tty_audit.so disable=* enable=root

Table 3: Reviewing keystroke captures with aureport

/etc/pam.d/system-auth pam_tty_audit addition

#aureport --tty -ts today

10 SELinux is developed by the National Security Agency (NSA), Red Hat, and the open source community.

11 The cron daemon is used to execute scheduled commands.
12 tty – serial terminal lines.
13 PAM is a system of libraries that handle the security tasks of applications on the system. The library provides a stable application

programminginterfacethatprivilegedgrantingprogramsdefertoforspecificsecuritytasks.

Using an open source framework to catch the bad guy

www.redhat.com 7

NIST SP800-92 notes that an audit log is a record of the events occurring within a system or network.
Logsarecomposedoflogentries.Eachlogentrycontainsspecificinformationrelatedtoaneventthathas
occurred.Logsshouldbeusedinconjunctionwithothernetwork/computerlogfilestopaintacomplete
story/history of an occurrence. As an example, a Network Intrusion Detection device might detect an attack
signature against a particular RHEL host or even record malicious commands given from a particular server.
Investigate using the audit tools; the RHEL host audit logs may indicate further evidence if a particular user
wasloggedintothehostatthatspecifictime,andifthespecificattackwassuccessful.

TheRHEL6AuditingSubSystemallowsthehosttogranularlylogandtrackusers,accesstofiles,direc-

tories, as well as system resources and system calls. Real-time monitoring can locate occurrences of or
accumulation of these auditable events that may indicate an imminent violation of security policy. Red Hat
hasalsohardenedtheauditlogfilesagainstloginjectionattacksbecauseallentrustedfieldshavebeen
formatted in hex encoded ascii to allow correct parsing. The RHEL 6 audit capabilities enable an organiza-
tion to monitor a system for application misbehavior or code malfunctions. By creating a management policy

consistingofasophisticatedsetofrulesincludingfilewatchesandsystemcallauditing,anorganizationcan
make sure that any violation of its security policies are noted and properly addressed.

With the increasing number of threats and the number and volume of computer security logs ever on the

increase, there is true demand for Computer Security Log Management. Log Management is the process
for generating, transmitting, storing, and analyzing computer security log data. The following sections
cover each of these aspects in depth and map the RHEL 6 audit sub system to being used in an operational
process.

1.1 estaBlishing policies and procedures for log management

To establish and maintain successful log management activities, an organization should develop a standard

process for performing log management. Most security policies state that there should be testing and audit
bytheInformationSystemsSecurityOfficer(ISSO)

and/or Information Systems Security Manager (ISSM)

of the security posture of the information system by employing various intrusion/attack detection and
monitoringtools.Theoutputofsuchtoolsmustbeprotectedagainstunauthorizedaccess,modification,or
deletion. These tools must also build upon audit reduction and analysis tools to aid the ISSO/ISSM in the
monitoring and detection of suspicious, intrusive, or attack-like behavior.

Anorganizationshoulddefineitsloggingrequirementsandgoals.Dependingonthethetypeofbusiness

or organization these requirements and goals could be very different. In addition to these requirements
andgoals,anorganizationshouldthendeveloppoliciesthatdefinelogmanagementauditactives.Log
Managementensuresthatcomputersecurityrecordsarestoredinsufficientdetailforanappropriate
period of time. Routine use of RHEL 6 audit tools to review and analyze will identify security incidents, policy
violations, and fraudulent activity in real-time. Table 4: Red Hat Enterprise Linux audit tools
provides an overview of the tools in RHEL 6. This along with the other fundamentals of Log Management
covered in this whitepaper are useful in performing forensic analysis as well as supporting the organization’s
internal investigations.

14 ISSO in this context could also mean the responsibility of the System Administrator as well in some organizations.

Using an open source framework to catch the bad guy

8 www.redhat.com

taBle 4: red hat enterprise linux audit tools

aUDIt tool

DEfINItIoN

auditd

The daemon auditd is the user space component of the Linux Auditing System. It is
responsibleforwritingauditrecordstothedisk.Configuringtheauditrulesisdone
with the auditctl utility; during start-up, the rules in the /etc/audit/audit.rules are
read by the auditctlcommand.Theauditdaemoncanbecustomizedinthefile
/etc/audit/auditd.conf. Viewing the logs is done with the ausearch, aureport,
and aulast facilities.

/etc/audit/audit.rules

The audit.rulesfilecontainsauditrulesthatwillbeloadedbytheauditdaemon’sinit
script any time the daemon is started. The auditctl program is used by the initscripts to
performactionsinthisfile.

/etc/audit/auditd.conf

The auditd.conffileistheconfigurationfilefortheauditdaemon.

auditctl

The auditctl command is used to assist controlling the kernel’s audit system. You
can get status, and add or delete rules into kernel audit system. You can also use this
commandtosetawatchonafile.

ausearch

The ausearch command is used to query the audit daemon logs based for events based
on different search criteria.

aureport

The aureport command will produce a summary reports of the audit system logs.

aulast

The aulast command will print out a listing of the last logged in users similarly to the
program last and lastb. The aulast command searches back through the audit logs
orthegivenauditlogfileanddisplaysalistofallusersloggedinandoutbasedonthe
range of time in the audit logs.

autrace

The autrace audit tool is a program that will add the audit rules to trace a process
similar to strace

. The is very useful to see what a program maybe doing.

audispd

The audispd daemon is an audit event multiplexor. It has to be started by the audit
daemon in order to get events. It takes audit events and distributes them to child
programs that want to analyze events in real time.

/etc/audisp/audispd.
conf

The audispd.conffilecontrolstheconfigurationoftheauditeventdispatcher.

1.1.1 rhel 6 log storage and rotation

TheRHEL6AuditSubSystemallowsforthestorageoflogfilesfromboththesystemlevelandinfrastruc-

turelevel.Auditlogfilescanberetainedonthesystemaswellastransmittedtothelogmanagementinfra-
structure host. If either the system or infrastructure logging host fails to log, this allows the other to retain
the log data. In addition, during an incident on a system, the system’s logs might be altered or destroyed by
attackers. Incident response can then use the data from the infrastructure logs to help with the forensics.

Comparing both infrastructure logs to the system logs can also help to determine what data was changed or
removed, helping indicate what the attacker wanted to conceal.

SystemlogfilesinRHEL6arestoredinthe/var/log directory. This directory should have its own parti-

tion or logical volume

. The RHEL 6 audit sub system stores its logs in the /var/log/audit directory. This

should also have its own partition or logical volume. We recommended that both /var/log and var/log/
audit have their own separate partitions or logical volumes to keep the log data separate and secure. The
audit trail is so important in a CAPP environment (which supports many Regulations and Standards)

that

access to system resources must be denied if an audit trail cannot be created.

15 The strace command will trace system calls and signals. It is shipped in the strace RPM with RHEL 6.
16 ItisrecommendedthatpartitioningrequirementsshouldmatchtheUnitedStatesGovernmentsConfigurationBaseline(USGCB).Formore

information see http://usgcb.nist.gov/

17 PCI, FISMA, HIPAA, SOX, DOD Directive 8500.2, DCID 6/3 as examples.

Using an open source framework to catch the bad guy

www.redhat.com 9

Note: For the examples in this whitepaper, we will assume that CAPP must be met in our SecurityPolicy,
soallsettingshereinwillreflectthisassumption.

Thepartitionsandspecificconfigurationfilescanbesetafterthesystemhasbeeninstalledorwhenitis

provisioned and written within a kickstart

file.Akickstartfileallowsforautomation,whichprovides:

• Reliability: settings are performed in the same (correct) way every time.

Adedicatedpartitionpreventstheauditdlogsfromdisruptingsystemfunctionalityiftheyfillandprevents

any other activity in the /varfilesystemfromfillingthepartitionandstoppingtheaudittrail.

The partition size should be larger than the maximum space that auditd will use. The following formula

can help the system administrator determine the partition size. Where MAX_SIZE_OF_LOG_FILE is the
sizeofeachlogfile,andNUMBER_OF_LOG_FILESisthenumberoflogfilesbeingrotated(SeeTable 5:

Formula to determine log space).

taBle 5: log rotation script

foRMUla

MAX_SIZE_OF_LOG_FILE X NUMBER_OF_LOG_FILES

1.1.1.1 non auditdlogfilesin/var/log

Since we have assumed that we are using a CAPP environment, log rotation should be set system-wide. This
includeslogfilesthatauditddoesnotmanage.Logfilesin/var/log should be rotated as well as turning
compression on to save space on the system. This whitepaper is covering auditd in terms of forensics and
incident response, but it would not be complete if we did not take a section to cover all non auditdlogfiles
in /var/log as part of the rotation procedure.

Torotateaswellascompresstheselogfilesin/var/log, run the following script noted in Table 6: Log

rotation script. This script will set the log rotation to 12 weeks (3 months) and compress each

/var/loglogfile.Thefile/etc/logrotate.confisdesignedtoeaselogfileadministrationforthese

logfilesbyallowingautomaticrotationandcompression.Notethatinthe/etc/logrotate.conffileno
packages own wtmp and btmp,sotheyarerotatedinthisfile.RPMpackagesdroplogrotationinformation
into the /etc/logrotate.d directory. Also in RHEL 6 by default the dateext option is now enabled. This
optionarchivesoldversionsoflogfilesbyaddinganextensionrepresentingthedateinYYYYMMDD format.
Previously,anumberwasappendedtofiles.

taBle 6: log rotation script

SCRIPt

for logconf in `ls -1 /etc/logrotate.conf`
do
perl -npe ‘s/rotate\s+4/rotate 12/’ -i $logconf
perl -npe ‘s/\#compress/compress/’ -i $logconf
done

18 Kickstarts allow for an automated installation method where partitions as well as logical volumes can be set. Security settings discussed

inthiswhitepaperforauditcanalsobesetinakickstartfile.

Using an open source framework to catch the bad guy

10 www.redhat.com

1.1.1.2 auditdlogfilesinthedefault/var/log/auditdirectory

The default settings with auditd rotates 4 logs by size (5MB), retaining a maximum of 20MB of data. This

makes it possible to loose audit data with auditd.Justlikerotatingandcompressinglogfilesin /var/

log rotation and compression should also be done for the auditddaemon.Specificallythisisdoneinthe/

etc/logrotate.d/auditfile.ThebelowscriptinTable 7: Log rotation script for auditd sets
compression for audit compress and rotates the logs for 90 days (rotate 90). The log is kept daily (daily),
it will not rotate if it is empty (notifempty), and if the log is missing, go on to the next one without issuing
an error message (missingok). The lines between postrotate and endscript are executed using bash, in
this example restarting the audit daemon.

Itshouldbenotedthatcompressingthelogfileswillmaketheaudittoolsaureport and ausearch unable to
readthem.Ifyouusethesetoolswhenthelogfilesarecompressed,youwillhavetousethezcat or bzcat

commandstodecompressthefilestostdout for the audit tools to read into stdin. This will allow the ISSO

toworkwiththelogfileincompressedmode.

taBle 7: log rotation script for auditd

SCRIPt

cat «lOGROT1 > /etc/logrotate.d/audit
compress
/var/log/audit/audit.log
{
rotate 90
daily
notifempty
missingok
postrotate
/sbin/service auditd restart 2> /dev/null > /dev/null || true
endscript
}
lOGROT1

Itisalsoimportanttosettherotationtimetobeasclosetomidnightaspossible,sothatlogfilescanbe
rotated on a near daily basis according to the 24 hour clock. To do this, in the /etc/audit/auditd.conf
filesetthemax_log_file_action to ignore (See Table 8: max_log_file_action Setting and as an
alternative to doing the edit by hand, a script can also be run. See Table 9: max_log_file_action
Script):

taBle 8:max_log_file_actionSetting

SEttING

max_log_file_action=ignore

19 Both zcat and bzcatwilluncompressthelogfilesbyexaminingthecorrectmagicnumberwhethertheyhavethecorrect .gz or .bz2

suffixornot.Forspecificsonmagicnumberspleasereadtheman page for magic (5).

Using an open source framework to catch the bad guy

www.redhat.com 11

Next copy the script auditd.cron that was shipped with the audit RPM (located in /usr/share/doc/
audit-version directory) to the /etc/cron.daily directory, change the permissions to 0770, and make
sure the ownership remains root.root. See Table 10: auditd.cron script. After the rotate the log
will be named audit.log.1.

taBle 9: max_log_file_actionScript

SCRIPt

#perl -npe ‘s/max_log_file_action = ROTATE/max_logfile_action = IGNORE/’ /etc/audit/auditd.
conf

taBle 10: auditd.cronscript

SCRIPt

#!/bin/sh
##########
# This script can be installed to get a daily log rotation
# based on a cron job.
##########
/sbin/service auditd rotate
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t auditd “ALERT exited abnormally with [$EXITVALUE]”
fi
exit 0

RHEL 6 includes the cronie package as a replacement for vixie-cron. The main difference between these
packages is how the regular jobs (daily, weekly, and monthly) are done. Cronie uses the /etc/anacrontab

filetostartitsdailycron jobs, which is different from vixie-cron. To ensure that the daily rotation is close

to the 24 hour clock, you will want to edit /etc/anacrontabfilewiththefollowingchanges(SeeTable

11: /etc/anacrontab file). In the /etc/anacrontabfilethesettingsareasfollows:theRANDOM_DELAY
variable to 0 so no random delay is added, as well as setting the START_HOURS_RANGE to 0,whichdefines

the midnight interval when scheduled jobs can run. Lastly, we set the delay in minutes for cron.daily
to 0,sothatitspecifiesthatanacron will not delay and do cron.daily as close to midnight as possible.

Using an open source framework to catch the bad guy

12 www.redhat.com

taBle 11: /etc/anacrontabfile

/etc/anacrontab file

# /etc/anacrontab: configuration file for anacron
# See anacron(8) and anacrontab(5) for details.
SHell=/bin/sh
PaTH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=0
# the jobs will be started during the following hours only
START_HOURS_RANGE=0
#period in days delay in minutes job-identifier command
1 0 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly

1.1.2 rhel 6 remote host storage

TheRHEL6hostshouldbeconfiguredtosenditslogstoaremotehostaswellashavingalocalrepository.

An intruder or malicious user who has compromised the root accountonamachinemaydeletethelogfiles.

If system logs are to be useful in detecting malicious activities, it is necessary to send them to a remote log
server that is running defense-in-depth security features to protect the logs. Virtual Lans (VLANs) should
alsobeconsidered:havingthelogfilesbedistributedonaseparatenetwork.VLANsallowanorganization
to separate network segments and apply access control based on security rules. It is recommended that the
audit data be segmented on its own VLAN. This will also increase network performance and segment the
audit data over the network.

1.1.2.1 log management with the /etc/audit/auditd.conf file

As discussed and a continuous theme of this whitepaper, the purpose of auditing is being able to do an inves-

tigation periodically or whenever an incident occurs. Logs contain records of system and network security,
thustheyneedtobeprotectedfrombreachesoftheirconfidentialityandintegrity.Logsthataresecured
improperly in storage or in transit might also be susceptible to intentional or unintentional alteration and

destruction. This could cause activities to go unnoticed, and even hide the evidence to conceal the identity of
a malicious party.

Inadditiontotheconfidentialityandintegrityofarchivedlogfiles,organizationsneedtoalsoprotectthe
availabilityofthelogfiles.Forexample,logfilesizelimitandlogrotationplayimportantrolesforLog
Management in terms of data retention requirements. Table 12: The /etc/audit/auditd.conf log
server setup configurationdescribestheconfigurationsettingsfortheauditdaemonforaserverthat
isbeingconfiguredtoaggregateandcollectlogfilesfromnumeroushosts.Itisimportanttomakesurethat
theconfigurationissettoallowlogfilestogrowwithoutbound.

Using an open source framework to catch the bad guy

www.redhat.com 13

taBle 12: The/etc/audit/auditd.conflogserversetupconfiguration

aUDIt toolS

DEfINItIoN

log_file = /var/log/audit/
audit.log

Specifiesthefullpathnametothelogfilewhereauditrecordswillbestored.
Thismustbearegularfile.
Inthisexamplewechosethedefaultlogfile.

log_format = raw

Describes how the information should be stored on disk. There are two options:
raw and nolog. If set to raw, the audit records will be stored in a format exactly
as the kernel sends it. If the option is set to nolog, then all audit information is
discarded and not written to disk. This mode does not affect data sent to the
audit event dispatcher.

log_group = isso

Specifiesthegroupthatisappliedtothelogfile’spermissions.Thedefaultis
root. The group name can be either numeric or spelled out. This is the opportu-
nitytomakeagroupforalltheInformationSystemInformationOfficers.
In this example we assume that there is a group made for the Information
SystemsSecurityOfficerswhowillbelookingatthelogfiles.

priority_boost = 4

This is an non-negative number that tells the audit daemon how much of a
priority boost it should take.
The default is 4, which we are using in this example.

flush = data

Valid values are none, incremental, data, and sync.

none:nospecialeffortismadetoflushtheauditrecordstodisk.

incremental: If set to incremental, then the freq parameter is used to determine
howoftenanexplicitflushtodiskisissued.

data: The data parameter tells the audit daemon to keep the data portion of the
diskfilesync’datalltimes.

sync: The sync option tells the audit daemon to keep both the data and meta-
data fully sync’d with every write to the disk.
In this example, we are having the audit daemon to keep the data portion of the
diskfilesync’datalltimes.

freq =

This is a non-negative number that tells the audit daemon how many records to
writebeforeissuinganexplicitflushtodiskcommand.Theflushkeywordmust
be set to incremental.

num_logs = 90

Specifiesthenumberoflogfilestokeepifrotateisgivenasthemax_log_
file_action.Thisnumbermustbe99orless.Thedefaultis0whichmeansno
rotation.

disp_qos = lossless

Controls blocking/lossless or non-blocking/lossy communication between the
audit daemon and the dispatcher. There is a 128k buffer between the audit
daemon and dispatcher. If lossy is chosen, incoming events going to the
dispatcher are discarded when this queue is full. Lossy is the default value.

dispatcher = /sbin/audispd

The dispatcher program is a program that is started by the audit daemon when
it starts it. It will pass a copy of all audit events to that application’s stdin.
It this example we are using the dispatcher /sbin/audispd this will be set on
theserversandclientstoreceiveandsendlogfilestoanaggregatehost.

(continued on next page)

Using an open source framework to catch the bad guy

14 www.redhat.com

aUDIt toolS

DEfINItIoN

name_format = numeric

Controls how computer node names are inserted into the audit event stream.

none: no computer name is inserted into the audit event.
hostname: name returned by the gethostname syscall.
fqd: means that it takes the hostname and resolves it with dns for a fully quali-
fieddomainnameofthatmachine.
numeric: is similar to fqd, except it resolves the IP address of the machine.
user: userisanadmindefinedstringfromthenameoption.Thedefaultvalue
is none.

In this example we are setting each log to have the IP address of the host where
it originated. The IP address is inserted into the audit stream.

name =

Theadmin-definedstringthatidentifiesthemachineifuserisgivenasthe
name_format option.

max_log_file =

Specifiesthemaxfilesizeinmegabytes.Whenthelimitisreached,itwilltrigger
aconfigurableaction.Mustbeanumericvalue.
In this example we are not setting a max_log_file size.
We are rotating daily.

max_log_file_action = ignore

This parameter tells the system what action to take when the system has
detectedthatmaxfilesizelimithasbeenreached.

ignore: The audit daemon does nothing.
syslog: Issue a warning to syslog.
suspend: will cause the audit daemon to stop writing records to the disk.
rotate: causes the audit daemon to rotate the logs.
keep_logs: similar to rotate except it does not use the num_logs setting. This
prevents the audit logs from being overwritten.
In this example we are not setting a max_log_file size. We are rotating daily.

action_mail_acct = isso_
name@example.com

Contains a valid email address or alias. The default address is root. Requires /
usr/lib/sendmail to exist on the machine.
In this example, we have set the email address to a user named isso_name @
example.com. Of course this would have to be set to a valid user. Perhaps
different ISSOs will be monitoring different machines, so this can get granular.
RedHatNetworkSatelliteisagreatoptiontoprovidespecificfilestoagroupof
hosts. We could version control the /etc/audit/auditd.conffile.

space_left = 500

This is a numeric value in megabytes that tells the audit daemon when to
performaconfigurableactionbecausethesystemisstartingtorunlowon
space.
In this example, we have set a numeric value of 500 megabytes that will tell the
audit daemon to send an email as noted in space_left_action = email.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 15

aUDIt toolS

DEfINItIoN

space_left_action = email

This parameter tells the system what action to take when the system has
detected that it is starting to get low on disk space.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email:sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.

In this example we are saying that when the system is running low on disk space
at the 500 megabyte notice, email a warning to isso_name@example.com

admin_space_left = 200

This is a numeric value in megabytes that tells the audit daemon when to
performaconfigurableaction.Thisshouldbeconsideredthelastchancetodo
something before running out of disk space.
In this example, we have set a numeric value of 200 megabytes that will tell
the audit daemon to send an email as noted in admin_space_left_action =
email. note: This should be considered a last chance to do something before
running out of disk space.

admin_space_left_action =
email

This parameter tells the system what action to take when the system has
detected that it is low on disk space.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email: sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.
In this example we are saying that when the system is running low on disk space
at the 200 megabyte notice, email a warning to isso_name@example.com

disk_full_action = halt

This parameter tells the system what action to take when the system has
detectedthatthepartitiontowhichlogfilesarewrittenhasbecomefull.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email:sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.

In this example, we are assuming that the system is a system that is critical to
our security policy (as should be an aggregate log server) and when the /var/
log/audit partition is full, the machine will shut down.

(continued on next page)

Using an open source framework to catch the bad guy

16 www.redhat.com

aUDIt toolS

DEfINItIoN

disk_error_action = email

This parameter tells the system what action to take when there is an error
detected while writing audit events to disk or rotating logs.
ignore: nothing happens.
syslog: means it will issue a warning to syslog.
email:sendsaemailwarningtotheemailaccountspecifiedinaction_mail_acct
as well sending the message to syslog.
exec: /path/to/a/script
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.
In this example we are saying that when the system provides a disk error
notice, email a warning to isso_name@example.com We have elected to just
send an email because the system may still be able to write to disk, and that this
is something a systems administrator should look at.

tcp_listen_port = 60

This s a numeric value in the range of 1 – 65535, which causes auditd to listen
on the corresponding TCP port for audit records from remote systems.

The audit daemon can be linked with tcp_wrappers. Access controls can be set in
the /etc/hosts.allow and /etc/hosts.denyfiles.
In this example, we are having auditd listen on port 60 for incoming audit logs
from client servers. SELinux as well as iptables policy is established for port 60.

tcp_listen_queue = 200

This is a numeric value that indicates how many pending (requested but unac-
cepted) connections are allowed. The default is set to 5.
In this example we have adjusted the value to the number of systems on the
network that will be sending audit log data to the server. This is to ensure that if
we had all servers provisioned and /or started at the same time our connections
would not be rejected.

tcp_max_per_addr = 1

This is a numeric value that indicates how many concurrent con- connections
from one IP address is allowed. The default is 1 and the maximum is 16.
In this example we leave the default to one. The default should be adequate in
most cases unless a custom written recovery script runs to forward unsent
events. In this case you would increase the number only large enough to let it in
too.

use_libwrap= yes

This setting has a value of either yes or no. It determines whether or not to use
TCP wrappers to discern connection attempts that are from allowed machines.

Inthisexampleweareusingtcpwrappersfortheaddedsecuritybenefits.For
specificuseoftcpwrappersman page 5 host_access.

tcp_client_ports = 1-1023

This parameter may be a single numeric value or two values separated by a
dash. It indicates which client ports are allowed for incoming connections. If
notspecified,anyportisallowed.Valuesmaybe1-65535.Specifying1-1023
makes sure that clients send from a privileged port to help prevent log injection
attacks by untrustworthy users.

tcp_client_max_idle = 120

This parameter indicates the number of seconds that a client may be idle before
auditd complains.
In this example we set the number of seconds that a client may be idle to 120
before auditd complains. This is set higher than the client heartbeat_timeout
setting by a factor of two.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 17

aUDIt toolS

DEfINItIoN

enable_krb5 = yes

RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForwardsection
ofthiswhitepapertosendlogfilesencrypted.
If set to “yes,” Kerberos 5 will be used for authentication and encryption. The
default is “no.”
In this example due to security policy and the sensitivity of the information in
thelogfiles,wedonotwanttosendmessagesinthewireincleartext.Wewill
be using Kerberos 5 for encryption.

krb5_principal = auditd

RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForwardsection
ofthiswhitepapertosendlogfilesencrypted.
This is the principal for this server. The default is “auditd.”
In this example, we are using the default auditd as the Kerberos prin-
ciple.Giventhisdefault,theserverwilllookforakeynamedlikeauditd/
hostname@EXAMPLE.COM stored in  /etc/audit/audit.key to authenti-
cate itself, where hostname is the canonical name for the server’s host, as
returned by a DNS look up of its IP address.

krb5_key_file = /etc/audit/
audit.key

RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForwardsection
ofthiswhitepapertosendlogfilesencrypted.
Locationofthekeyforthisclient’sprinciple.Thekeyfilemustbeownedbyroot
and mode set to 0400. The default is /etc/audit/audit.key
Inthisexample,weareusingthedefaultkeyfile/etc/audit/audit.key. It
should be noted that there may be a name schema that is developed for the key
names, which would include a version control system. Red Hat Network Satellite
is an excellent way to safely store, version control, and provision this audit.key.

1.1.2.1.1 encryption over the wire with ssh port forwarding

RHEL 6 currently does not support Kerberos encryption with auditd (noted rows in the Tables with the color
dark gray). To implement this very important feature, this section uses SSH Port Forwarding as an alterna-
tiveexample.Logfileswhentransmittedmusthaveintegritymechanismsadequatetoassuretheintegrity
andconfidentialityofalltransmittedloginformation.Thisincludesthepreventionofhijackingofacommu-
nications session. SSH

Port Forwarding allows a port from one host to appear on another, using a connec-

tion through SSHandallowingthetrafficthatisbeingforwardedtobeencryptedwithSSH. The tunnel
provided with SSH Port Forwarding uses the TCP transport method. The supported ciphers are:

• 3des-cbc

• aes128-cbc

• aes192-cbc

• aes256-cbc

• aes128-ctr

• aes192-ctr

• aes256-ctr

• arcfour128

• arcfour256

20 SSH provides a secure encrypted communications between two hosts over an insecure network. With SSH you can specify which cipher to

use.

Using an open source framework to catch the bad guy

18 www.redhat.com

• arcfour

• blowfish-cbc

• cast128-cbc

As long as the remote aggregated log server is running the ssh daemon, it is possible to tunnel the auditd

logfilesovertothehostviassh. Tunneling uses SSH to create connections between the client RHEL 6
hostsendingitslogfilestotheaggregatedlogserver.Theclienthostmustspecifyanon-standardportto
connect. We have also set in both the /etc/auditd/auditd.conffile(seeTable 12: The /etc/audit/
auditd.conf log server setup configuration) and the /etc/audit/audisp-remote.conf (see

Table 16: /etc/audit/audisp-remote.conf log client setup configuration) to send outgoing
clientlogfilesandlistenforincomingclientlogfilesontheaggregatedlogserveronportslowerthanport
1023. Specifying ports 1-1023 makes sure that clients send form a privileged port to help prevent log injec-

tion attacks by untrustworthy users.

On the aggregated log server the ISSO will need to set up a port forwarding channel that listens for connec-

tions on the localhost. See Table 13: Aggregated log server port forwarding script. The script
sets up a SSHtunnelbetweenanaggregateloghostanditsclientsthatwillbesendingtheirlogfilestothe
server.Wefindthatitiseasiertocontrolandmanagethescriptfromtheaggregatelogserverandhaveit
run to the clients rather than having all the clients run a script (especially if the clients are in the hundreds).

Note: The script could also be run from /etc/init.d/add-log-server.sh. The script should handle the
start, stop, and status input commands. The ISSO could then use chkconfig –add add-log-server.
sh to set the script to be launched during booting. The script of course needs to be able to work with

iptables.

Using an open source framework to catch the bad guy

www.redhat.com 19

The script originates the SSH tunnel from the central aggregate log host machine and connects to the client

machinesthatitwillgetthelogfilesfrom.Thescriptshouldbecalledfrom/etc/rc.local so that it is run
every time the aggregate log hosts boots up and after all the network services have been run.

Examining the agg-log-server.sh script, the for command does the bulk of the work pulling from vari-
ables for the host names that the ISSO should set. The -R 61:loghost.example.com:60 initiates the
reverse SSH tunnel from Port 61 on the remote server to loghost.example.com Port 60 on the aggre-
gate log host machine. The -nflagtellsSSH to associate the standard input with /dev/null. There will
not be any command line input with SSH,justatunneltoencryptthelogfilesfromtheclienttotheaggre-
gate log server. The script also sends the standardoutput as well as the standarderror to /dev/null as
well (> /dev/null 2>&1). The -N option tells the SSH client to only set up the tunnel and do not prepare
a command stream for issuing commands on the remote system. The -T argument does not allocate a
pseudo-tty on the remote system. The -x argument disables X11 forwarding, just as a defense-in-depth
option.

Thescriptiseasytoimplement,anditachievesveryimportantsecuritygoalsforbothconfidentialityand

integrityofthelogfilesbeingsenttotheaggregatelogserver.Itshouldalsobenotedthattherearemany
ways to achieve this goal and that this is just an overview example. For example, we recommend using
Red Hat Network Satellite to distribute scripts to client systems (whether virtualized or bare metal),
keepingconfigurationmanagementbestpracticesaswellasmaintainingsecurityonthescriptitself.

taBle 13: aggregated log server port forwarding script

SCRIPt

#!/bin/bash
#Red Hat Federal Senior Solutions Architect Team
#Written By: Norman Mark St. Laurent
#Name: agg-log-server.sh
#Version: 1.1
#Summary:
# This script will setup a SSH tunnel between an aggregate log host
# and its clients that will be sending their log files. The script
# will originate the SSH tunnel from the central aggregate log
# host machine, and connect to the client machines that it will get
# the log files from.
# The Script should be called from /etc/rc.local so that it gets run
# every time it boots up.
#
#Last Modified: 12/06/2011

#Aggregate Log Server Hostname
HOSTNAME1=$(echo $HOSTNAME | awk -F. ‘{ print $1 }’)
#Array of Hostnames for Centralized Logging
HOSTNAMECLIENTS=( $HOSTNAME1 hostname1 hostname2 hostname3 hostname4 )

for CHOSTS in ${HOSTNAMECLENTS[@]}
do
/usr/bin/ssh -nNTx -R 61:loghost.example.com:60 $CHOSTS.example.com >
/dev/null 2>&1
done

Using an open source framework to catch the bad guy

20 www.redhat.com

1.1.2.2 Logmanagementwiththe/etc/audisp/audispd.conffile

The /etc/audisp/audispd.conffileisthefilethatcontrolstheconfigurationoftheauditeventdispatcher.
Thisfileshouldbesetuponboththeaggregatelogserveraswellastheclientsasitwillcontroltheremote

server setup and actions needed by the audit event multiplexer audispd. It takes audit events and distrib-
utes them to child programs that want to analyze events in real time. See Table 14: The /etc/audisp/
audispd.conf filefortheconfigurationsettingforthisfile.

taBle 14: the /etc/audisp/audispd.conffile

aUDIt RUlE

DEfINItIoN

q_depth = 400

This is a numeric value that tells how big to make the internal queue of the
auditeventdispatcher.Abiggerqueueletsthedispatcherhandleafloodof
events better. If syslog indicates that audit events are getting dropped, then
increase this number. The default value is 80.

overflow_action

Thisoptiondetermineshowthedaemonshouldreacttooverflowingits
internal queue. When this happens, it means that more events are being
received than it can get rid of.
ignore: nothing happens.
syslog:means it will issue a warning to syslog.
suspend: will cause the audit daemon to stop writing records to the disk.
single: will put the system in single user mode.
halt: will shutdown the computer system.

priority_boost

This is a non-negative number that tells the audit event dispatcher how much
of a priority boost it should take. The default is 4. No change is 0.

max_restarts

This is a non-negative number that tells the audit event dispatcher how many
times it can try to restart a crashed plug-in. The default is 10.

name_format

This is the option that controls how computer node names are inserted into
the audit event stream.
none: no computer name is inserted into the audit event. This is the default.
hostname: is the name returned by the gethostname syscall.
fqd: means that it takes the hostname and resolves it with dns for a fully quali-
fieddomainnameofthemachine.
numeric: is similar to fqd except it resolves the IP address of the machine.
user: istheadminuserdefinedstringfromthenameoption.

name

Thisistheadmindefinedstringthatidentifiesthemachineifauserisgivenas
the name_format option.

1.1.2.2.1 Logmanagementwiththe /etc/audit/audisp-remote.conffile

ToconfigureaRHEL6clienthostforremoteloggingtoanaggregateloggingserver,youmustusethe

audisp-remote plugin for the audit event dispatcher daemon audispd. The ISSO can tell if the audisp-
remote plugin is installed by running the following RPM command. See Table 15: RPM command for
audisp-remote plugin. If the rpm is not installed, then install it with the yum

command.

21 YUM(YellowdogUpdaterModified)isaninteractive,RPM based, package manager written at Duke University. It can automatically perform

system updates, including dependency analysis and obsolete processing based on “repository” metadata.

Using an open source framework to catch the bad guy

www.redhat.com 21

taBle 15: rpm command for audisp-remote plugin

CoMMaND

[root@mstlaure /]# rpm -qa | grep audispd-pugins
audispd-plugins-2.1-5.el6.x86_64

taBle 16: /etc/audit/audisp-remote.conflogclientsetupconfiguration

ExaMPlE SEttING

DESCRIPtIoN

remote_server = 192.168.1.22

This is a one word character string that is the remote server hostname or
IP Address that this plugin will send log information to. This can be the
numeric address or a resolvable hostname.

port = 60

This option indicates what port to connect to on the remote log server.
In this example, we are having auditd on the aggregate log host listen on
port 60 for incoming audit logs from client servers. SELinux as well as
iptables policy is established for port 60.

local_port = 61

This option indicates what local port to connect from on the local machine.
You can use the option any, which will set the port to any available unprivi-
leged port. The port should be set to a used port less than 1024, like we
did in this example. This ensures that only privileged users can bind to that
port.
Ifyousetaspecificport,thenyouwillhavetomatchtheportnumberon
the aggregating auditd.conffiletcp_client_ports directive to match
the ports that the client is sending from.
Inthisexample,wewanttosendthelogfileviaaprivilegedport.Thisis
important to ensure that only privileged users can bind to that port. This
matches the settings in the tcp_client_ports in the aggregating auditd.
conffileontheserver.

transport = tcp

This parameter tells the remote logging plugin how to send the events to
the remote system. The only valid option currently is tcp. If this is set to
tcp, the remote logging plugin will make a normal clear text connection to
the remote system.
This is not used if Kerberos is enabled.

If an ISSO is aggregating multiple machines, that person should enable node information in the audit event
stream. This can be done in one of two places. If computer node names are written to disk as well as sent
to the realtime event stream, then edit the name_format option in the /etc/audit/auditd.conffile

(See Table 12: /etc/audit/auditd.conf log server setup configuration) as an example. If the

security requirements need to have node names only appear in the realtime event stream, then edit the
name_format option in the /etc/audisp/audispd.conf. If you edit in both places then it will put two
nodefieldsintheeventstream.

Table 16: /etc/audit/auditd.conf log client setup configuration describes the needed setting

in the /etc/audit/audisp-remote.conf filethatwillallowaRHEL6hosttobecomealogclient,which
willthensenditslogfilesencryptedoverthewiretoaRHEL6aggregatelogserveronthenetwork.The

/etc/audit/audisp-remote.conffilecontrolstheconfigurationoftheauditremoteloggingsubsystem.

(continued on next page)

Using an open source framework to catch the bad guy

22 www.redhat.com

ExaMPlE SEttING

DESCRIPtIoN

mode = forward

This parameter tells the remote logging plugin what strategy to use getting
records to the remote system. Valid values are immediate and forward.
If set to immediate, the remote logging plugin will attempt to send events
immediately after getting them.
In this example we set to forward. The plugin will then store the events
to disk and then attempt to send the records. If the connection cannot be
made, it will queue the records until it can connect to the remote system.
The depth of the queue is controlled by the queue_depth option.

queue_file = /var/log/audit/
remote.log

Thisisthepathofthefilethatisusedfortheeventqueueifmodeissetto
forward. The default is /var/spool/audit/remote.log.
Inthisexample,wehaveoptedtoputthelogfileinthe/var/log/audit
directory because of it residing on its own partition and for easy mainte-
nance with aide (which we will talk about in Part 2 of this whitepaper).

queue_depth = 400

This option determines how many records can be buffered to disk or in
memory before considering to be a failure sending. The default is 200.
It this example for security purposes we have upped this number to 400.
In RHEL 6.2 the default is 120 (which is low). Each slot eats about 9Kof
memory.

network_retry_time = 5

The time, in seconds between retires, when a network error is detected.
The default is 1 second. This applies after the second attempt to avoid
unneededdelaysinareconnectissufficienttofixtheproblems.

format = managed

This parameter tells the remote logging plugin what data format will be
used for the messages sent over the network. The default is managed,
which adds some overhead to ensure each message is properly handled on
the remote end, and to receive status messages from the remote server.
If ascii is given instead, each message is a simple ASCII text line with no
overhead.
If mode is set to forward, the the format must be managed.

max_tries_per_record = 10

The maximum number of times an attempt is made to deliver each
message. The minimum value is 1; the default is 3. If too many attempts
are made, the network_failure_action is performed.
In this example, we set the max_tries_per_record to 10.

heartbeat_timeout = 60

This parameter determines how often in seconds the client should send a
heartbeat event to the remote server. This is used to let both the client and
server know that each end is alive and has not terminated.
The default value is 0 which disables sending a heartbeat. In this example,
we have asked for a heartbeat every 1 minute.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 23

ExaMPlE SEttING

DESCRIPtIoN

network_failure_action = suspend

This parameter tells the system what action to take whenever there is an
error detected when sending audit events to the remote system. Values
are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec:/path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shutdown the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

disk_low_action = suspend

Likewise, this parameter tells the system what action to take if the remote
end signals a disk low error. The default is to ignore it. Values are:
ignore:  the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single:  this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt:  this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

disk_full_action = suspend

Likewise, this parameter tells the system what action to take if the remote
end signals a disk full error. The default is to ignore it. Values are:
ignore:  the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single:  this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt:  this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

(continued on next page)

Using an open source framework to catch the bad guy

24 www.redhat.com

ExaMPlE SEttING

DESCRIPtIoN

disk_error_action = suspend

Likewise, this parameter tells the system what action to take if the remote
end signals a disk error. The default is to ignore it. Values are:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

remote_ending_action = suspend

Likewise, this parameter tells the system what action to take if the remote
end signals a disk full error. The default is to ignore it. Values are:
ignore:  the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec:/path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop:this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt:  this option will cause the remote logging plugin to shut down the
computer system.
reconnect:  this option tells the remote plugin to attempt to reconnect to
the server upon receipt of the next audit record. If it is unsuccessful, the
audit record could be lost. The default is to suspend logging.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 25

ExaMPlE SEttING

DESCRIPtIoN

generic_error_action = syslog

Likewise, this parameter tells the system what action to take if the remote
end signals an error we do not recognize.. The default is to send it. To
syslog. Values are:
ignore:  the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single:  this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt:  this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

generic_warning_action = syslog

Likewise, this parameter tells the system what action to take if the remote
end signals a warning we do not recognize.. The default is to send it. To
syslog. Values are:
ignore:  the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single:  this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt:  this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

(continued on next page)

Using an open source framework to catch the bad guy

26 www.redhat.com

ExaMPlE SEttING

DESCRIPtIoN

queue_error_action = suspend

Likewise, this parameter tells the system what action to take if there is a
problem working with a local record queue. The default is to send it. To
syslog. Values are:
ignore:  the remote logging plugin does nothing.
syslog:means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single:  this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt:  this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

overflow_action = syslog

Likewise, this parameter tells the system what action to take if the internal
eventqueueoverflows.Thedefaultistosendit.tosyslog.Valuesare:
ignore: the remote logging plugin does nothing.
syslog: means that it will issue a warning to syslog.
exec: /path/to/a/script that will be executed.
suspend: causes the remote logging plugin to stop sending records to the
remote system. The logging plugin will still be alive.
single: this option will cause the remote logging plugin to put the computer
system in single user mode.
stop: this action will cause the remote logging plugin to exit, but leave
other plugins running.
halt: this option will cause the remote logging plugin to shut down the
computer system.
In this example, we have opted to suspend. This causes the remote logging
plugin to stop sending records to the remote system. The logging plugin
will still be alive.

enable_krb5 = yes

RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
If set to “yes,” kerberos 5 will be used for authentication and encryption.
The default is “no.”
In this example do to security policy because of the sensitivity of the infor-
mationinthelogfiles,wedonotwanttosendmessagesonthewireinclear
text. We will be using Kerberos 5 for encryption.

krb5_principal =

RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
This is the principal for this server. The client and server will use the speci-
fiedprincipletonegotiatetheencryption.
Inthisexample,wedonotusefiled.Novariableisspecifiedsothekrb5_
client_name and remote_server values are used.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 27

ExaMPlE SEttING

DESCRIPtIoN

krb5_client_name = auditd

RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
Thisspecifiesthenameportionoftheclient’sownprincipal.Ifunspeci-
fied,thedefaultisauditd. The remainder of the principal will consist of
thehost’sfullyqualifieddomainnameandthedefaultKerberosrealm:
auditd/host14.example.com@EXAMPLE.COM (assuming you gave
“auditd” as the krb_client_name).
Note that the client and server must have the same principal name and
realm.

krb5_key_file = /etc/audit/audit.
key

RHEL6currentlydoesnotsupportKerberos.SeeSSHPortForward
sectionofthiswhitepapertosendlogfilesencrypted.
Locationofthekeyforthisclient’sprincipal.Notethatthekeyfilemust
be owned by root and mode 0400. The default is /etc/audisp/audisp-
remote.key
Inthisexample,weareusingthedefaultkeyfile/etc/audit/audit.key.
It should be noted that there maybe a name schema that is developed for
the key names, which would include a version control system.
Red Hat Network Satellite

is an excellent way to safely store, version

control, and provision this audit.key.

1.1.3 specific rhel 6 log generation settings

As we saw in Section 1.1.1 and Section 1.1.2logsourcesneedtobeconfiguredsothattheycapture

the necessary information in the desired format as well as the desired locations, and retain information for
theappropriateperiodoftime.RHEL6allowsforitslogsourcestohaveverygranularconfigurationoptions
and allows an organization to meet CAPP requirements.

By default, the audit daemon auditd only logs SELinux denials, which are helpful for monitoring SELinux
anddiscoveringintrusionattemptsandsecurityeventssuchasmodificationstouseraccountsandcalls
to sudo.Inthissection,theaudit.rulesfilewillbetunedtoprovidegranulardatainthelogfiles.Themost
current data for the auditd logs is stored in the /var/log/audit/audit.logfile.Therotatinglogfiles
are also found in the directory /var/log/audit.Rotationnamesinthisdirectoryfortherotatedfilesare

/var/log/audit/audit.log.{1,2,3,n}.Logfilesarestoreddailyinoursetup.Thus,onedayback

would be audit.log.1, two days back would be audit.log.2 and so on. When using the aureport,
ausearch, and aulast commands remember to use the --input file-nameflagtouseaspecificlogfile
other than the default /var/log/audit/audit.log. This is to aid in analysis where the logs have been
moved or rotated.

The auditd daemon in RHEL 6 is responsible for writing audit records to the disk. During start-up, the rules

in /etc/audit/audit.rules are read by auditd. Editing the /etc/audit/audit.rulesfileallowsforthe
configurationofspecificpolicy.Viewingtheauditlogsisdonewiththeausearch, aureport, and aulast
commands. The ISSO will want to make sure that the auditd service is enabled. This is the default, but
should be checked (See Table 17: Enabling auditd command).

22 Red Hat Network (RHN) Satellite is a systems management platform that makes Red Hat Linux deployable, scaleable, manageable,

and consistent. See http://www.redhat.com/red_hat_network/ for more information on RHN Satellite.

Using an open source framework to catch the bad guy

28 www.redhat.com

If any of the run levels were noted to be “off” the ISSO should start the auditd to run at all run levels (See

Table 18: Enabling auditd run levels).

taBle 17: enaBling auditd command

CoMMaND

[root@mstlaure /]# chkconfig --list | grep auditd
auditd

0:off 1:off 2:on

3:on

4:on

5:on

6:off

taBle 18: enaBling auditd run levels

CoMMaND

[root@mstlaure /]# chkconfig auditd on ; service auditd start
Starting auditd:

To ensure that all processes can be audited, even those which start prior to the audit demon, add the

argument audit=1 to the kernel line in the /etc/grub.conffile.Youcanuseyourfavoriteeditortodoso
(See Table 19: /etc/grub.conf edit to start audit) or an automated shell script (See Table 20:

/etc/grub.conf audit automated script).

taBle 19: /etc/grub.confedittostartaudit

KERNEl lINE: /EtC/GRUb.CoNf ExaMPlE

kernel /vmlinuz-2.6.32-131.17.1.el6.x86_64 ro root=/dev/mapper/HelpDeskRHEL6-Root rd_
LVM_LV=HelpDeskRHEL6/Root rd_LUKS_UUID=luks-82b2a0a6-c7e5-4219-8b4d-2e8fed1418c4 rd_
LVM_LV=HelpDeskRHEL6/Swap rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16
KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet vga=0x318 audit=1

Automated one liner to get the job done.

taBle 20: /etc/grub.confauditautomatedscript

SCRIPt

#sed -i ‘/^kernel/s|$| audit=1|’ /boot/grub/grub.conf

1.1.3.1 Logmanagementwiththe/etc/audit/audit.rulesfile

The audit.rulesfilegoeshandinhandwiththeauditctl command. The auditctl command is a utility

used to assist controlling the kernel’s audit sub system. It can be used to control the behavior, get status,
and add or delete rules into the 2.6 kernel’s audit sub system. The rules for maintainability should be kept in
the /etc/audit/audit.rulesfile.Thisfilesisessentiallythesamewhentypinginanauditctl command
at the shell prompt except that the actual command name “auditctl: is not needed as it is implied. The
audit rules come in three varieties (See Table 21: The /etc/audit/audit.rules varieties).

Using an open source framework to catch the bad guy

www.redhat.com 29

taBle 21: the /etc/audit/audit.rules varieties

aUDIt toolS

DEfINItIoN

Control

Controlcommandsgenerallyinvolveconfiguringtheauditsystemrather
than telling it what to watch for. These commands include: deleting all rules
on start-up, setting the size of the kernels backlog queue, setting the failure
mode, setting the event rate limit. These rules are usually given at the top of
the audit.rulesfile.

File System

Thefilesystemrulesarealsocalledwatches.Watchesareusedtoaudit
accesstoparticularfilesanddirectories.Ifthepathgivenintheruleisa
directory, then the rule is used recursively to the bottom of the directory tree
(excluding any directories that maybe mount points).

The syntax is as follows:
RHEL4andRHEL5

-w /path/to/file/to/watch -p permissions -k keyname
RHEL6Recommended
-a exit,always -F path=/path/to/file/to/watch -F key=keyname

Where the permissions are anyone of the following:

r-readofthefile
w–writetothefile
x–executethefile
a-changeinthefile’sattribute

System Call

The system call rules are loaded into a matching engine that intercepts each
syscall that all programs on the system make.
This can effect performance. The more rules, the bigger the performance
hit. You can help performance by combining syscalls into one rule whenever
possible.

Thekernelhasfiefilters:

Task: Only checked the fork or clone syscall.
Entry:Runs through each syscall entry.
Exit:Checkedonthesyscallexit.TheEntryfilterwillbedeprecated.Rules
ontheexitfilteraremuchmorecommonandallfieldsareavailableforuseat
syscall exit.
user:Usedtofiltereventsthatoriginateinuserspace.Fieldsthatarevalid
for use are uid, auid, gid, and the pid.
Exclude: Used to exclude certain events from being emitted.

Using an open source framework to catch the bad guy

30 www.redhat.com

Using keys in both the watches and system call rules to give the rule a meaning helps to reduce the workload
and make the policy granular.

The RHEL 6 audit sub system should collect the execution of privileged commands for all users and the root

user to meet CAPP requirements. This requires adding an audit rule to watch execution of each setuid

setgid

program. The following command will locate all setuid and setgid programs for each local parti-

tion on the system. The lines found would then need to be added to the /etc/audit/audit.rulesfile.
PARTequalsthePartitiontosearchwiththefindcommand(e.g.,“/home” or “/”) (See Table 22: Finding
and setting setuid and setgid watches).

The auditd program can perform comprehensive monitoring of system activity. This section describes

configurationsettingsforcomprehensiveauditingusingtheDepartmentofDefenseDefenseInformation
SystemsAgencySecurity(DISA)TechnicalImplementationGuide(STIG)asthefoundationandbasisforthe
audit rules that are set in the example /etc/audit/audit.rulesfile.Theauditsubsystemsupportsan
amplecollectionofevents,toincludethetracingofarbitrarysystemcalls(identifiedbysystemcallname,or

bysystemcallnumber.TheauditsubsystemcanalsofilterbyPID,UDI,systemcallsuccess,andsystemcall

argument.Perhapstheauditsubsystem’smostgranularaspectistheabilitytomonitoryspecificfilesfor
modificationstothefile’scontentsormetadata.

TheNationalSecurityAgency(NSA)hasalsodevelopedanddistributedconfigurationguidanceforRHEL

that is currently being used throughout the government and by numerous entities as a security baseline for
theirRHELsystems.Youwillnotetheseauditrecommendationsinthe/etc/audit/audit.rulesfileexample
below (See Table 23: The /etc/audit/audit.rules file). Table 23: The /etc/audit/audit.

rules filealsoprovidesadefinitionofeachsettingsothattheISSOhasadeepunderstandingoftherule
aswellasabasistochangeforaspecificrequirementthataspecificorganizationmayhave.

Notes:

Thesettingshereinmaybechangeddependingonyourorganization’sspecificauditpolicy.Thesettingsare

are for use as a best case scenario to get you started and to follow CAPP.

The audit rules and settings in this whitepaper follow the recommended audit rules settings in /usr/share/

doc/audit-version/stig.rules

. Red Hat has provided audit.rules templates that meet a number of

standards and regulalations. These can also be found in this directory. We recommend starting with the
stig.rules and modifying as need be.

The system `arch` lines will have to be removed if they do not match the system that the /etc/audit/

audit.rulesfileison.

23 The setuid(ShortforsetuserID)isanaccessrightsflagthatallowuserstorunanexecutablewiththepermissionsoftheexecutables

owner.

24 The setgid(ShortforsetgroupID)isanaccessrightsflagthatallowuserstorunanexecutablewiththepermissionsoftheexecutables

group.

25 TheSTIG(SecurityTechnicalImplementationGuide)isamethodologyforstandardizedsecureinstallationofRedHatEnterpriseLinux.

TheDefenseInformationSystemsAgency(DISA)cratestheconfigurationguidanceinsupportoftheDOD.

Using an open source framework to catch the bad guy

taBle 22: finding and setting setuid and setgid watches

SCRIPt

#find PART -xdev $ -perm 4000 -o -perm -2000 $ \
-type f | awk ‘{ print “-a always,exit -F path=” $1 “ -F perm=x \
-F auid>=500 -F auid!=4294967295 -k ISSO-privileged” }’

www.redhat.com 31

taBle 23: the /etc/audit/audit.rulesfile

aUDIt RUlE

DEfINItIoN

-D

Remove any existing rules.

-b 8192

Increase the buffer size to handle the increased number of
messages.

-f 2

Set the failure mode to panic. This option lets you deter-
mine how you want the kernel to handle critical errors.

-e 2

Thisparameterwilllocktheauditconfigurationsothat
it cannot be changed. This should be the last line in the
audit.rulesfile.

-a exit,always -F path=/var/log/audit -F
key=ISSO-audit

Successful and unsuccessful attempts to read information
fromtheauditrecords;allmodificationstotheaudittrail.

a exit,always -F path=/etc/audit -F perm=wa -F
key=ISSO-cfg-audit

-a exit,always -F path=/etc/sysconfig/auditd -F
perm=wa -F key=ISSO-cfg-audit
-a exit,always -F path=/etc/libaudit.conf -F
perm=wa -F key=ISSO-cfg-audit
-a exit,always -F path=/etc/audisp/ -F perm=wa
-F key=ISSO-cfg-audit

Modificationstoauditconfigurationthatcanoccurwhile
theauditcollectionfunctionareoperating;allmodifica-
tions to the set of audited events.

This portion will allow enough audit information to deter-
mine the date and the time of the action, the locale of the
action, the system entity that initiated or completed the
action, the resources involved, and the action involved.

-a always,exit -F arch=b32 -S settimeofday -S
stime -F key=ISSO-time-change
-a always,exit -F arch=b64 -S settimeofday -F
key=ISSO-time-change
-a exit,always -F path=/etc/localtime -F
perm=wa -F key=ISSO-time-change

Changes for RHEL 6.2
-a always,exit -F arch=b32 -S clock_settime -F
a0=0 -F key=ISSO-time-change

-a always,exit -F arch=b64 -S clock_settime -F
a0=0 -F key=ISSO-time-change

Things that could affect time.

(continued on next page)

Using an open source framework to catch the bad guy

32 www.redhat.com

aUDIt RUlE

DEfINItIoN

-a always,exit -F arch=b32 -S sethostname -S
setdomainname -F key=ISSO-system-locale
-a always,exit -F arch=b64 -S sethostname -S
setdomainname -F key=ISSO-system-locale
-a exit,always -F path=/etc/issue -F perm=wa
-F key=ISSO-system-local
-a exit,always -F path=/etc/issue.net -F
perm=wa -F key=ISSO-system-locale
-a exit,always -F path=/etc/hosts -perm=wa -F
key=ISSO-system-locale
-a exit,always -F path=/etc/sysconfig/network
-F perm=wa -F key=ISSO-system-local

Things that could affect system locale.

-a exit,always -F path=/etc/selinux -F perm=wa
-F key=ISSO-mac-policy

Things that could affect Mandatory Access Control (MAC)
policy.

-a always,exit -F arch=b32 -S chmod -S fchmod
-S fchmodat -F auid>=500 -F auid!=4294967295
-F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S chmod -S fchmod
-S fchmodat -F auid>=500 -F auid!=4294967295
-F key=ISSO-dac-policy
-a always,exit -F arch=b32 -S chown -S
fchown -S fchownat -S lchown -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S chown -S
fchown -S fchownat -S lchown -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b32 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500
-F auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b32 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy
-a always,exit -F arch=b64 -S setxattr -S
lsetxattr -S fsetxattr -S removexattr -S
lremovexattr -S fremovexattr -F auid>=500 -F
auid!=4294967295 -F key=ISSO-dac-policy

Things that could affect Discretionary Access Control
(DAC) permissions.

-a exit,always -F path=/var/log/faillog -F
perm=wa -F key=ISSO-logins
-a exit,always -F path=/var/log/lastlog -F
perm=wa -F key=ISSO-logins
-a exit,always -F path=/var/log/tallylog -F
perm=wa -F key=ISSO-logins
-a exit,always -F path=/var/log/faillock -F
perm=wa -F key=ISSO-logins

Successful and unsuccessful logins and logoffs. The
Red Hat login, gdm, and openssh programs will log all
relevant information.
The faillock command is an application that can be used
toexamineandmodifythecontentsofthetallyfiles.It
can display the recent failed authentication attempts of
theusernameorclearthetallyfilesforallorindividual
usernames.Itusesthe/var/log/faillockfiletorecord
information.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 33

aUDIt RUlE

DEfINItIoN

Successful and unsuccessful accesses to security relevant
objects and directories, including creation, open, close,
modification,anddeletion.

-a exit,always -F path=/var/run/utmp -F
perm=wa -F key=ISSO-session
-a exit,always -F path=/var/log/btmp -F
perm=wa -F key=ISSO-session
-a exit,always -F path=/var/log/wtmp -F
perm=wa -F key=ISSO-session

Session initiation is audited by pam without any rules
needed.
The utmpfileletsyoudiscoverinformationaboutwhois
currently using the system.

-a always,exit -F arch=b32 -S creat -S
open -S openat -S truncate -F exit=-
EACCES -F auid>=500 -F auid!=4294967295 -F
key=ISSO-access
-a always,exit -F arch=b32 -S creat -S open -S
openat -S truncate -F exit=-EPERM -F auid>=500
-F auid!=4294967295 -F key=ISSO-access
-a always,exit -F arch=b64 -S creat -S
open -S openat -S truncate -F exit=-
EACCES -F auid>=500 -F auid!=4294967295 -F
key=ISSO-access
-a always,exit -F arch=b64 -S creat -S open -S
openat -S truncate -F exit=-EPERM -F auid>=500
-F auid!=4294967295 -F key=ISSO-access

Unsuccessfulaccessattemptstofiles

-a always,exit -F path=/bin/ping -F perm=x
-F auid>=500 -F auid!=4294967295 -F
key=ISSO-privilege

Use of privileged commands (unsuccessful and successful)

-a always,exit -F arch=b32 -S mount
-F auid>=500 -F auid!=4294967295 -F
key=ISSO-media-export
-a always,exit -F arch=b64 -S mount
-F auid>=500 -F auid!=4294967295 -F
key=ISSO-media-export

Export to media (successful)

-a exit,always -F path=/etc/sudoers -F perm=wa
-F key=ISSO-admin-actions

Record system administration actions

-a exit,always -F path=/etc/group -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/passwd -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/gshadow -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/shadow -F perm=wa
-F key=ISSO-auth
-a exit,always -F path=/etc/security/opasswd
-F perm=wa -F key=ISSO-auth

Changes in user authentication and identity.

-a exit,always -F path=/var/log/audit/audit.
log -F key=ISSO-audit-logs

Audit trail protection. The contents of the audit trail
shouldbeprotectedagainstunauthorizedaccess,modifi-
cation,ordeletion.Thisiscoveredbycorrectfilepermis-
sions, but activity watches are set here.

(continued on next page)

Using an open source framework to catch the bad guy

34 www.redhat.com

aUDIt RUlE

DEfINItIoN

-a always,exit -F arch=b32 -S ptrace -F
key=ISSO-tracing
-a always,exit -F arch=b64 -S ptrace -F
key=ISSO-tracing

Could indicate hacker activity or just a programmer
debugging.

-a always,exit -F arch=b32 -S personality -k
ISSO-bypass
-a always,exit -F arch=b64 -S personality -k
ISSO-bypass

Could be an attempt to bypass audit, or simply a legacy
program.

-a exit,always -F path=/sbin/insmod -F perm=x
-F key=ISSO-modules
-a exit,always -F path=/sbin/rmmod -F perm=x
-F key=ISSO-modules
-a exit,always -F path=/sbin/modprobe -F
perm=x -F key=ISSO-modules
-a always,exit -F arch=b32 -S init_module -S
delete_module -F key=ISSO-modules
-a always,exit -F arch=b64 -S init_module -S
delete_module -F key=ISSO-modules

Module actions (insert,delete, and probes).

-a exit,always -F path=/etc/cron.deny -F
perm=wa -F key=ISSO-cron-at-jobs
-a exit,always -F path=/etc/cron.allow -F
perm=wa -F key=ISSO-cron-at-jobs
-a exit,always -F path=/etc/at.deny -F perm=wa
-F key=ISSO-cron-at-jobs
-a exit,always -F path=/etc/at.allow -F
perm=wa -F key=ISSO-cron-at-jobs

Specificwatchesaresethereforinternalprogramfiles.

1.2 red hat enterprise linux 6 log management operational process

Whatfollowsisthemostimportantsectionofthiswhitepaper.Nowthatwehaveconfiguredauditdto
collectlogsaccordingtoCAPP,theremustbeanoperationalprocessinplacetoreviewthelogfiles.This
processmustberoutine,organized,flexible,andmademandatorywithintheorganization’ssecuritypolicy.
Otherwise,alltheconfigurationandguidanceweappliedintheprevioussectionareirrelevantandthereis
no framework to catch the “BadGuy.” The following sections put forth guidelines on an operational process

thatmonitorslogfilesinrealtime.

1.2.1 defining roles and responsiBilities

NISTSpecialPublication800-92“GuidetoComputerSecurityLogManagement”notesanorganization
shouldperformsignificantplanningandpreparatoryactionsforperforminglogmanagementandtoestab-
lish and maintain successful a log management infrastructure. It is recommended that the ISSO oversee the
log management infrastructure as well as analyzing the logs periodically, reporting on the results of the log
management activities to the ISSM. In addition, system and network administrators need to periodically
analyzethelogfiles.Securityadministratorsifavailableshouldalsoperformloganalysis.

Typically, system, network, and security administrators are responsible for managing logging on their

systems, performing regular analysis of their log data, as well as documenting and reporting the results. For
the purpose of this whitepaper we will call the subset of these individuals ISSOs.

Using an open source framework to catch the bad guy

www.redhat.com 35

1.2.2 rhel 6 forensics and incident response log analysis

When performing log analysis or even working a postmortem investigation, an ISSO should start up front
with the main aureport output to just get an idea about what is happening on the system. This report will

tell you about events that are hard coded by the audit system such as login and logout, uses of authentica-
tion, system anomalies, how many users have been on the machine, and if SELinux has detected any AVCs

Once a point of interest has been found, the ISSO can look up the event with the ausearch -a event

number (as all reports have the audit event number). Specifying start and stop times will also help narrow

downspecifics.Thereportsproducedbytheaureport command should be used as building blocks for
more complicated analysis. See Table 25: Log analysis commands by shift for a detailed review and
example of forensics and incident response log analysis. This table provides a starting point and routine to

helpfindanomaliesandsituationsthatdonotcomplywiththesecuritypolicy.Useittoevolvebestprac-
ticesinyourdailyauditreviewingactivities.Youwillbesurprisedwhatyoufind,andtheinformationwillput

you one step ahead of the BadGuy!

taBle 25: log analysis commands By shift

SHIft

loG aNalYSIS CoMMaND

DESCRIPtIPoN

1st Shift:
6:00am – 2:00pm

The 1st Shift is the Primary Shift
reviewinglogfilesfromthedaybefore
as well as generating reports.

aureport --summary --start
yesterday

Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the previous day.

The 1st Shift should run reports
summarizing the last days events.

Use ausearch --event audit-
event-id if need be to tunnel down.

aureport --failed --start
yesterday

Running this report will allow the ISSO
to get statistics of failed events.

(continued on next page)

In the next section we setup a routine log analysis policy geared toward using auditd on a RHEL 6 system.
Because we have setup remote host logging, we have chosen in this example to perform log analysis
centrally.CentralizinglogfilesallowstheISSOtogetthebigpicture,asithaseverylogfilefromtheenter-
prisecentrallyontheaggregateloghost.RedHataudittoolscanbeusedtotunneldownandfindspecific
notesofinterest.Remembertousethe–inputfile-nameflagwiththetoolstoviewdatainarotatedlogfile.

Youcouldalsotakeasubsetoflogfilesandconcatenatethemtogetherwiththecat command for analysis

(togathermultiplerotationdaysoflogfiles).See(Table 24: Concatenating compressed log files
in the /var/log/audit directory for analysis). This command actually will concatenate all the log
filesinthe/var/log/audit directory. The ISSO could creative by using the seqcommandtopullaspecific

number of logs out.

26 SELinux AVC (Access Vector Cache) is a new operating system component that provides caching of access decision computations to

minimize the performance overhead of the Flask security mechanisms.

Using an open source framework to catch the bad guy

taBle 24: Concatenatingcompressedlogfilesinthe/var/log/auditdirectoryforanalysis

SCRIPt

for i in `ls -1 /var/log/audit/audit.log.*` ; do cat /var/log/audit/audit.log > reviewlog.audit ; zcat $i >> reviewlog.audit ;
done

36 www.redhat.com

SHIft

loG aNalYSIS CoMMaND

DESCRIPtIPoN

aureport –l --failed --start
--yesterday

This command will allow the ISSO
to get more granular detail of failed
events for login-related events.

aureport –f --failed --start
--yesterday

This command will allow the ISSO
to get more granular detail of failed
eventsforfile-relatedevents.

aureport –p --failed --start
--yesterday

This command will allow the ISSO
to get more granular detail of failed
events for process-related events.

aureport –u --failed --start
--yesterday

This command will allow the ISSO
to get more granular detail of failed
events for user-related events.

aureport –k KeyName --start
--yesterday

This command will provide a high level
report on all the keys we set in the
audit.rulesfile.SpecificKeyName
we can use are all the Keys starting
with our ISSO-KeyName.

ausearch -k ISSO-audit --start
--yesterday

This command will allow the ISSO to
see both successful and unsuccessful
attempts to read information from the
auditrecordsandanymodifications
to the audit trail.

ausearch -k ISSO-cfg-audit
--start --yesterday

This command will allow the ISSO
toseemodificationstoaudit
configuration.

ausearch -k ISSO-time-change
--start --yesterday

This command will allow the ISSO to
see any audit record that could affect
the time of the system.

ausearch -k ISSO-system-local
--start --yesterday

This command will allow the ISSO to
see any audit record that could note a
change in system locale.

ausearch -k ISSO-MAC-policy
--start --yesterday

This command will allow the ISSO to
see any audit record that could note
a change in system the Mandatory
Access Control Policy.

ausearch -k ISSO-MAC-policy
--start --yesterday

This command will allow the ISSO to
see any audit record that could note
a change in system the Mandatory
Access Control Policy.

ausearch -k ISSO-access
--start --yesterday

This command will allow the ISSO to
see any audit record that could note
unsuccessfulaccessattemptstofiles.

ausearch -k ISSO-privilege
--start --yesterday

This command will allow the ISSO to
see any use of privileged commands
both unsuccessful and successful.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 37

SHIft

loG aNalYSIS CoMMaND

DESCRIPtIPoN

ausearch -k ISSO-media-export
--start --yesterday

This command will allow the ISSO to
see any and all successful exports to
media.
The ISSO should make note of the
user.

ausearch -k ISSO-admin-actions
--start --yesterday

This command will allow the ISSO to
seeallactionwiththesudoersfile.

ausearch -k ISSO-auth --start
--yesterday

This command will allow the ISSO to
see all changes in user authentication
and identity.

ausearch -k ISSO-audit-logs
--start --yesterday

This command will allow the ISSO to
see if any unauthorized access, modi-
fication,ordeletionhastakenplace
on the audit trail.

ausearch -k ISSO-modules
--start --yesterday

This command will allow the ISSO to
see if any unauthorized access, modi-
fication,ordeletionhastakenplace
with kernel modules.

ausearch -k ISSO-cron-at-jobs
--start --yesterday

This command will allow the ISSO
to see if any unauthorized access or
modificationhastakenplacewith
cron and at jobs (cron.deny, cron.
allow, at.deny,and at.allow).

aureport --summary --start
today 00:00:01

Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the days events to
the current time.
Use ausearch --event audit-
event-id if need be to tunnel down.

ausearch – a audit_event_id

Running this search will allow the
ISSO to view all records carrying a
suspicious audit event ID. Each audit
event message has a unique ID. One
application’s system call may have
several events that are logged, and
this will allow a trail of more than one
record to be pieced together to tell a
story.

(continued on next page)

Using an open source framework to catch the bad guy

38 www.redhat.com

SHIft

loG aNalYSIS CoMMaND

DESCRIPtIPoN

2nd Shift:
2:00pm – 10:00pm

aureport --summary --start
today 00:00:01

Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the day’s events
to the current time.

Use ausearch --event audit-
event-id to tunnel down for further
investigation.

This will also check for the morning
shift as well as yesterday, as most of
the organization's programmers work
during the morning shift.

ausearch -k ISSO-ptrace
--start --yesterday

This command will allow the ISSO to
see hacker activity by a user, or just a
programmer debugging.
Should be investigated.

ausearch -k ISSO-bypass
--start --yesterday

This command will allow the ISSO to
see if there was an attempt to bypass
audit, or it could be a legacy program.

Should be investigated. Any legacy
program that has been approved
should be noted as a false positive.

Could | grep -v to elininate false
positives.

3rd Shift:
10:00pm – 6:00am

aureport --summary --start
today 00:00:01

Running this report will allow the
ISSO to get a rough overview of the
current audit statistics (events, logins,
processes, etc.) for the day’s events
to the current time.

Use ausearch --event audit-
event-id to tunnel down for further
investigation..

aulast --bad --start today
00:00:01

Running this report will allow the ISSO
to report on all bad logins for the day.

All users found in this list should be
emailed and asked if they had failed
loginsforthatspecificday.When
they come in for work the next day,
they will see their email. Policy states
that they are to reply back if they did
not have the failed login attempt.

(continued on next page)

Using an open source framework to catch the bad guy

www.redhat.com 39

PaRt 2: HoSt-baSED INtRUSIoN DEtECtIoN SYStEM

A host-based IDS provides the data integrity needed to ensure adequate protection of information and

system data, helping you meet security requirements and compliance. In Red Hat Enterprise Linux 6, the
RPM program and AIDE program delivers continuous and automated monitoring for security compliance as
well as implementing the needed security controls for a true defense-in-depth approach allowing for Built-in
Forensics, Incident Response, and Security to catch the BadGuy.

The RPM Package Manager (RPM) is a program that can be used as a host-based IDS. RPM contains various

optionsforqueryingpackagesandtheircontents.Theseverificationoptionscanbeinvaluabletoaforensics
investigationandcouldleadtocriticalsystemfilesandexecutablesthathavebeenmodified.

AdvancedIntrusionDetectionEnvironment(AIDE)isafileintegritycheckertoolthatisshippedwithRed

Hat Enterprise Linux 6. Using rules read from the /etc/aide.conffile,AIDEcreatesadatabaseoffile
attributes and extended attribute information. It uses several hashing algorithms for integrity checking,
including md5, sha1, rmd10, tiger, haval, sha256, and sha512. Once the database is initialized it
canbeusedtoverifytheintegrityoffiles.

coming soon

SHIft

loG aNalYSIS CoMMaND

DESCRIPtIPoN

Additional investigation command
notes:

These commands may also
tunneldownandprovidespecific
information.

To see all syscalls made by a
specific program:
auditctl -a exit,always -S all
-F pid=1005

To see files opened by a
specific user:
auditctl -a exit,always -S
open -F auid=510

To see unsuccessful open
calls:
auditctl -a exit,always -S
open -F success=0

String based matches
(Hostname, IPADDR, Filename,
SELinux Context)
ausearch --word IPADDRESS
Search for an event with the
given login user ID.
ausearch –loginuid login-id

Using an open source framework to catch the bad guy

red hat sales and inquiries

north america
1–888–REDHAT1
www.redhat.com
sales@redhat.com

europe, middle east
and africa
00800 7334 2835
www.europe.redhat.com
europe@redhat.com

asia pacific
+65 6490 4200
www.apac.redhat.com
apac@redhat.com

latin america
+54 11 4329 7300
www.latam.redhat.com
info-latam@redhat.com

is the registered

trademark of Linus Torvalds in the U.S. and other countries.

www.redhat.com

#0000000_0010

BiBliography

1. Kent,Karen,andMurugiahSouppaya.GuidetoComputerSecurityLogManagementRecommendations

oftheNationalInstituteofStandardsandTechnology.Gaithersburg,MD:U.S.Dept.ofCommerce,
Technology Administration, National Institute of Standards and Technology, 2006. Print.

2. Mayer, Frank, Karl MacMillan, and David Caplan. SELinux by Example: Using Security Enhanced Linux.

Upper Saddle River, NJ: Prentice Hall, 2007. Print.

3. Common Criteria : The Common Criteria Portal. Web. 06 Dec. 2011. <http://www.commoncriteriaportal.

org/>.

4. TheUnitedStatesGovernmentConfigurationBaseline(USGCB)-NIST.Web.06Dec.2011.<http://usgcb.

nist.gov/index.html>.

5. Need,Business.“Redhat.com|Government.”Redhat.com|TheWorld’sOpenSourceLeader.Web.06

Dec. 2011. <http://www.redhat.com/solutions/government/>.

6. Frields, By Paul. “Red Hat Magazine | SSH Port Forwarding.” Red Hat Magazine | Now Showing: open-

source.com. Web. 06 Dec. 2011. <http://magazine.redhat.com/2007/11/06/ssh-port-forwarding/>.

7. MS. Http://csrc.nist.gov/groups/STM/cmvp/documents/140-1/140sp/140sp1384.pdf.

8. Gift,ByNoah.“RedHatMagazine|AdvancedSSHConfigurationandTunneling:We

Don’t Need No Stinking VPN software.” Red Hat Magazine | Now Showing: open-
source.com. Web. 06 Dec. 2011. <http://magazine.redhat.com/2007/11/27/
advanced-ssh-configuration-and-tunneling-we-dont-need-no-stinking-vpn-software/>.

9. ”RemoteLoggingwithSSHandSyslog-NG.”ComputerForensicInvestigationsandInformationSecurity

Consulting | Deer Run Associates. Web. 06 Dec. 2011. <http://www.deer-run.com/~hal/sysadmin/
SSH-SyslogNG.html>.

10. “Operating Systems - NSA/CSS.” Welcome to the National Security Agency - NSA/CSS. Web. 07 Dec. 2011.

<http://www.nsa.gov/ia/mitigation_guidance/security_configuration_guides/operating_systems.shtml>.
“GuidetotheSecureConfigurationOfRedHatEnterpriseLinux5”.

11. Information Assurance Support Environment Home Page. Web. 07 Dec. 2011. <http://iase.disa.mil/>.