Logfile Parser Application seeking advice

Lee_Page · July 25, 2013, 8:15am

Hi All

I have been tasked with a Xojo project and looking for some advice on best method to accomplish the following.

I need to parse my IIS webserver logfile (high transactional site) every 5 minutes to loop through search for and make a total count of a string “BJLive/Default.aspx?”
bearing in mind that the log file is anything up to 500MB in size.

Thanks

Lee

Alex_von_Siebenthal · July 25, 2013, 8:24am

Textinputstream.ReadLine (in combination with instr()) might be a good way to go. I believe it doesn’t load the whole file into memory, just one line at a time. Due to the lack of an example file of that size I can’t test speed.

Lee_Page · July 25, 2013, 8:27am

Thanks Alex do you know if there are any code snippets floating around somewhere to reference?

Thanks

Lee

Alex_von_Siebenthal · July 25, 2013, 8:34am

Have a look at the wiki page, there is one… http://documentation.xojo.com/index.php/TextInputStream.ReadLine

This might be a start (untested):

Dim f As FolderItem Dim textInput As TextInputStream Dim rowFromFile As String Dim count As Integer =0 f=GetOpenFolderItem("text/plain") //defined as a FileType If f <> Nil Then textInput =TextInputStream.Open(f) textInput.Encoding=Encodings.UTF8 Do if instr(0,textInput.ReadLine, "BJLive/Default.aspx?") > 0 then count = count + 1 end if Loop Until textInput.EOF textInput.Close End If

However, with this snippet and the use of instr() it is assumed, that the search string only occurs max once per line. You might need to adjust it if happens to be more times in one line.

Lee_Page · July 25, 2013, 8:37am

Alex thanks so much for your help starting to make sense.

Lee_Page · July 25, 2013, 10:02am

Can anyone help a little further please. I have a text label that needs to display the total count value.

Thanks

Wayne_Golding · July 25, 2013, 10:06am

Label1.text = Str(count)

Lee_Page · July 25, 2013, 10:07am

Thanks Wayne LOL how did i miss that…

Lee_Page · July 25, 2013, 11:09am

Almost there Just one more thing to work out.

I have the following code in a timer event running every 5 minutes.

[code]

dim d As New Date
dim datetime as string = str(d.year) + “-” + str(d.Month) + “-” + str(d.day) + " " +str(d.hour) + “:” + str(d.minute)
TextField1.text = datetime

dim datetimesearch as string = str(d.year) + “-” + str(d.Month) + “-” + str(d.day) + " " +str(d.hour) + “:” + str(d.minute - 5)
TextField2.Text = datetimesearch

Dim f As FolderItem
Dim textInput As TextInputStream
Dim rowFromFile As String
Dim count As Integer =0

f = GetFolderItem(“c:\ex130725.log”)
If f <> Nil Then
textInput =TextInputStream.Open(f)
textInput.Encoding=Encodings.UTF8
Do
if instr(0,textInput.ReadLine, “BJLive/Default.aspx?”) > 0 then
count = count + 1
Label1.text = Str(count)
end if
Loop Until textInput.EOF
textInput.Close
End If[/code]

So basically what i am trying to achieve is to read the log file and make a count of how many times an HTTP request has been made for webpage “BJLIVE/Default.aspx?” within last 5 minute intervals. (Getting the system date+time now minus 5 minutes)

Here is one line from the log file
2013-07-25 01:22:50 W3SVC1 CHI-PROD-LTW-1 172.24.50.30 GET /bjlive/Default.aspx

Thanks I really appreciate help given with this.

Alex_von_Siebenthal · July 25, 2013, 12:51pm

So what is your exact question?

Lee_Page · July 25, 2013, 1:21pm

Hi Alex just trying to work out how to read the first line in the log file get the date and time from the first line and return only the total count of instr(0,textInput.ReadLine, “BJLive/Default.aspx?” for the previous 5 minutes

2013-07-25 01:22:50 W3SVC1 CHI-PROD-LTW-1 172.24.50.30 GET /bjlive/Default.aspx // First Line Entry
evrerything else in between
2013-07-25 01:17:50 W3SVC1 CHI-PROD-LTW-1 172.24.50.30 GET /bjlive/Default.aspx // Last Entry up to a 5 minute period

This technique is new to me in Xojo

I presume that I may need to use some regular expression first and then parse the results.

Richard_Duke · July 25, 2013, 3:56pm

MID(Logline, 12,8)

Alex_von_Siebenthal · July 25, 2013, 3:57pm

Like Richard said - have a look at the Mid() function and the Date class

Lee_Page · July 25, 2013, 4:51pm

Thanks for your help Alex and Richard. Very much appreciated

Still learning on the Job. Xojo is awesome.

Tim_Hare · July 25, 2013, 5:10pm

The date and time in your file is in SQLDateTime format, so you can create a new Date object and assign that portion of the string to it directly. Then you can use the TotalSeconds property to determine if it is in range.

dim now as new Date dim logdate as new Date logdate.SQLDateTime = left(stringFromTheFile, 19) if now.TotalSeconds - logdate.TotalSeconds < 300 then // happened in the last 5 minutes end

Lee_Page · July 25, 2013, 6:20pm

Hi Tim,

???

It’s a text file just a IIS log file

Keith_Stone · July 25, 2013, 6:41pm

Yes, it’s a text file but the displayed format of the date/time matches SQLDateTime text format.

In any case I wouldn’t use the date/time. Depending on the traffic that’s not granular enough. Save the file position (ie. line count) from the previous run and move forward. You’re going to have to keep some data off to the side anyway since the file name will change every day usually at midnight local time or GMT depending on the configuration, so you’re going to need to parse the end of one file and the beginning of the next each day.

Lee_Page · August 10, 2013, 11:18am

thanks for all your help. Been in hospital so sorry for late response. Have some catching up to do.

How would subract 5 minutes from SQLDatetime?

dim d As New Date dim datetime as string = d.SQLDateTime

Thanks

Alex_von_Siebenthal · August 10, 2013, 12:16pm

Have a look at Totalseconds

Lee_Page · August 10, 2013, 1:56pm

Thanks Alex

d.Totalseconds = d.Totalseconds - 300