Technical Note: Transferring historical logs from a FortiGate hard disk to a FortiAnalyzer
Products
FortiAnalyzer
FortiGate
Description

This articles explains the steps required to move logs previously stored on a FortiGate Hard Disk to a FortiAnalyzer so that those logs can be included in FortiView or Reports. 

An example of this might be if purchasing a FortiAnalyzer after a FortiGate has been in production.

It describes using an open source tool called  lz4_reader on a Windows workstation. 

Notes:
1) You can use the same tool on a MAC or Linux workstation but need to chose the -jar option when running the executable and need to have JDK (Java Development Kit) installed.

2) If you download a log file from the FortiOS GUI, it will not be compressed in LZ4 format, thus bypassing the need to perform the conversion described in this tech note.

3) Please refer to 'Technical Note: Importing multiple logs into FortiAnalyzer' in the Related Articles on how to inject them all back in FortiAnalyzer if needed as one single file

Solution

The logs stored on the FortiGate Hard Disk are in format  LZ4 and can not be directly imported to the FortiAnalyzer without first making some modifications. 

It is necessary to translate the LZ4 logs files to txt format using a FortiGate tool called "lz4_reader".

Note: The tool is attached to this KB article for the convenience of readers.  It is provided "as is" and is not maintained by Fortinet.

1.- Export all logs from FortiGate Hard Disk to FTP server.

 

FGTXXXXXXXXXX034 (root) # execute backup disk alllogs ftp 192.168.10.100 ftptest ftptest

Please wait...

Connect to ftp server 192.168.10.100 ...

Sent log file tlog.65147 to ftp server as tlog_FGTXXXXXXXXXX034_root_20170421_020000 OK.

Please wait...

 

Connect to ftp server 192.168.10.100 ...

Sent log file elog.65129 to ftp server as elog_FGTXXXXXXXXXX034_root_20170421_020000 OK.

Please wait...

 

Connect to ftp server 192.168.10.100 ...

Sent log file plog.65438 to ftp server as plog_FGTXXXXXXXXXX034_root_20170421_001645 OK.

Please wait...

 

Connect to ftp server 192.168.10.100 ...

Sent log file rlog.65147 to ftp server as rlog_FGTXXXXXXXXXX034_root_20170421_020000 OK.

Please wait...

 

FGTXXXXXXXXXX034 (root) #


2.- Uncompress the "lz4_reader” log conversion tool.

Uncompress (using a tool like WinRAR)  "lz4_reader” (a 3rd party tool attached to this technote for convenience) into a path on a local PC.

In the example below, the path used is “C:\Users\MARK\Documents\lza_reader>”.

 note: The "lz4_reader" tools translate LZ4 logs to TXT format.  In the example outlined in this article, the tool was run in Windows 10 with Java v8 ( build 1.8.0_77-b03).

C:\Users\MARK\Documents\lza_reader>dir

El volumen de la unidad C es Windows

El número de serie del volumen es: 641A-5B1F

 

Directorio de C:\Users\MARK\Documents\lza_reader

 

27/04/2017 03:01 p. m. <DIR> .

27/04/2017 03:01 p. m. <DIR> ..

11/10/2016 12:48 p. m. 6,148 .DS_Store

11/10/2016 12:49 p. m. 4,096 ._.DS_Store

11/10/2016 12:47 p. m. 3,253,658 log_reader.jar

29/09/2016 01:27 p. m. 693 run.bat

4 archivos 3,264,595 bytes

2 dirs 1,701,749,608,448 bytes libres

 

C:\Users\MARK\Documents\lza_reader>

 

3.- Translate the LZ4 file into TXT format

·          In a CMD of Windows run the command “run” into the directory where is was uncompressed.

·          Choose the option 1

·          Type the complete FG log file path in your Windows PC

·          The tool “lz4_reader” will create a directory and will put all files changed to TXT into this path.

 

C:\Users\MARK\Documents\lza_reader>run

Please input command number and enter...

To read a log, enter 1

To terminate the reader, enter 2

1

Input the path of the log you want to read...

C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000

The path you input is C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000

All readable contents are saved to C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.

Presione una tecla para continuar . . .

Please input command number and enter...

To read a log, enter 1

To terminate the reader, enter 2

2

 

4.- Rename the file extension from “txt” to “log”

C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable> dir

El volumen de la unidad C es Windows

El número de serie del volumen es: 641A-5B1F

 

Directorio de C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable

 

27/04/2017 03:05 p. m. <DIR> .

27/04/2017 03:05 p. m. <DIR> ..

27/04/2017 02:59 p. m. 3,680,094 tlog_FGTXXXXXXXXXX034_root_20170421_020000

27/04/2017 03:05 p. m. 35,075,188 tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.txt

2 archivos 38,755,282 bytes

2 dirs 1,701,587,505,152 bytes libres

 

C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable>

C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable>

C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable> rename tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.txt tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log

 

C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable> dir

El volumen de la unidad C es Windows

El número de serie del volumen es: 641A-5B1F

 

Directorio de C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable

 

27/04/2017 03:09 p. m. <DIR> .

27/04/2017 03:09 p. m. <DIR> ..

27/04/2017 02:59 p. m. 3,680,094 tlog_FGTXXXXXXXXXX034_root_20170421_020000

27/04/2017 03:05 p. m. 35,075,188 tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log

2 archivos 38,755,282 bytes

2 dirs 1,701,659,672,576 bytes libres

 

C:\Users\MARK\Downloads\tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable>

 

5.- From the FortiAnalyzer CLI , import the txt file with extension .log by FTP

 

FAZVM64 # execute log import ftp 192.168.10.100 ftptest ftptest tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log FGTXXXXXXXXXX034

Do you want to continue? (y/n)y

 

Log Import Info: Connect to ftp server 192.168.10.100 ...

Log Import Info: Found 1 .log or .csv files in remote folder : tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log .

Log Import Info: 1 log files found in remote folder, MAX import file setting is 10000, so 1 files will be imported.

 

Log Import Info: Downloading files from 192.168.10.100 ...#

Log Import Info: Log file tlog_FGTXXXXXXXXXX034_root_20170421_020000_readable.log was successfully imported to FGTXXXXXXXXXX034/tlog.1492668005.log.

Log Import Info: 1 log files are imported.

Log Import Info:

1 files are processed, 0 files remain.

FAZVM64 #


Once the FortiAnalyzer has finished importing the logs into the SQL database, the logs will be visible in LogView and FortiView, and available during report generation.

Related Articles
Technical Note: Importing multiple logs into FortiAnalyzer
Attachments
lz4_reader.tar.gz
Last Modified Date: 07-05-2019 Document ID: FD40716