qmsk.irclogs: log_parser.py@a0662cff1d9d (annotated)

50 f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	1	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	2	Parse log data into log_events
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	3	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	4
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	5	import datetime
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	6
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	7	import log_line
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	8	from log_line import LogTypes
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	9
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	10	class LogParser (object) :
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	11	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	12	Abstract interface
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	13	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	14
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	15	def __init__ (self, tz, timestamp_fmt="%H:%M:%S") :
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	16	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	17	Setup the parser to use the given format for line timestamps, which are of the given timezone
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	18	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	19
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	20	self.tz = tz
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	21	self.timestamp_fmt = timestamp_fmt
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	22
64 cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	23	def parse_lines (self, lines, date=None, starting_offset=None) :
50 f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	24	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	25	Parse the given (iterable) lines of unicode text into a LogEvent, no trailing newline.
64 cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	26
cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	27	Offset is the starting offset, and may be None to not use it.
50 f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	28
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	29	Giving date lets the parser build full timestamps, otherwise, unless line timestamps have full date
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	30	information, event timestamps will have a date component of 1900/1/1.
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	31	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	32
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	33	abstract
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	34
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	35
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	36	class IrssiParser (LogParser) :
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	37	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	38	A parser for irssi logfiles
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	39	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	40
64 cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	41	def parse_lines (self, lines, date=None, starting_offset=None) :
50 f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	42	"""
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	43	Parse the given lines, yielding LogEvents.
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	44	"""
65 8b50694f841e improve search further Tero Marttila <terom@fixme.fi> parents: 64 diff changeset	45
64 cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	46	for offset, line in enumerate(lines) :
50 f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	47	# status lines
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	48	if line.startswith('---') :
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	49	# XXX: handle these
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	50	continue
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	51
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	52	# normal lines
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	53	else :
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	54	# XXX: only parse timestamps for now
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	55	timestamp, data = line.split(' ', 1)
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	56
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	57	# parse timestamp into naive datetime
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	58	dt = datetime.datetime.strptime(timestamp, self.timestamp_fmt)
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	59
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	60	# override date?
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	61	if date :
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	62	dt = dt.replace(year=date.year, month=date.month, day=date.day)
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	63
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	64	# now localize with timezone
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	65	dtz = self.tz.localize(dt)
f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	66
64 cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	67	# offset?
65 8b50694f841e improve search further Tero Marttila <terom@fixme.fi> parents: 64 diff changeset	68	if starting_offset :
64 cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	69	offset = starting_offset + offset
cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	70
cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	71	else :
cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	72	offset = None
cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	73
50 f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	74	# yield raw events
64 cdb6403c2498 beginnings of a LogSearchIndex class Tero Marttila <terom@fixme.fi> parents: 50 diff changeset	75	yield log_line.LogLine(offset, LogTypes.RAW, dtz, None, data)
50 f13cf27a360b implement more LogSource features (logs for date, cleanup last_logs), implement irssi parser, formatter, other misc. stuff Tero Marttila <terom@fixme.fi> parents: diff changeset	76

author	Tero Marttila <terom@fixme.fi>
	Tue, 10 Feb 2009 03:22:43 +0200
changeset 80	a0662cff1d9d
parent 65	8b50694f841e
child 83	a34e9f56ddda
permissions	-rw-r--r--