log_search.py
author Tero Marttila <terom@fixme.fi>
Tue, 10 Feb 2009 23:59:37 +0200
changeset 87 39915772f090
parent 74 1ab95857d584
child 89 2dc6de43f317
permissions -rw-r--r--
update LogSearchIndex to use new LogLine fields
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     1
"""
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     2
    Full-text searching of logs
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     3
"""
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     4
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     5
import datetime, calendar, pytz
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
     6
import os.path
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     7
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     8
import HyperEstraier as hype
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
     9
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    10
import log_line
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    11
74
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    12
class LogSearchError (Exception) :
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    13
    """
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    14
        General search error
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    15
    """
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    16
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    17
    pass
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    18
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    19
class NoResultsFound (LogSearchError) :
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    20
    """
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    21
        No results found
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    22
    """
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    23
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    24
    pass
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
    25
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    26
class LogSearchIndex (object) :
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    27
    """
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    28
        An index on the logs for a group of channels.
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    29
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    30
        This uses Hyper Estraier to handle searching, whereby each log line is a document (yes, I have a powerful server).
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    31
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    32
        These log documents have the following attributes:
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    33
            @uri            - channel/date/line
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    34
            channel         - channel code
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    35
            type            - the LogType id
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    36
            timestamp       - UTC timestamp
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    37
            source_nickname - source nickname
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    38
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    39
        Each document then has a single line of data, which is the log message itself
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    40
    """
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    41
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    42
    def __init__ (self, channels, path, mode='r') :
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    43
        """
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    44
            Open the database at the given path, with the given mode:
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    45
                r       - read-only
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    46
                w       - write, create if not exists
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    47
                a       - write, error if not exists
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    48
                c       - write, create, error if exists
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    49
                *       - write, create, truncate if exists
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    50
            
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    51
            Channels is the ChannelList.
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    52
        """
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    53
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    54
        # store
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    55
        self.channels = channels
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    56
        self.path = path
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    57
        self.mode = mode
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    58
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    59
        # check it does not already exist?
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    60
        if mode in 'c' and os.path.exists(path) :
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    61
            raise LogSearchError("Index already exists: %s" % (path, ))
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    62
        
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    63
        # mapping of { mode -> flags }
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    64
        mode_to_flag = {
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    65
            'r':    hype.Database.DBREADER,
67
13975aa16b4c fix LogSearchIndex open permissions
Tero Marttila <terom@fixme.fi>
parents: 66
diff changeset
    66
            'w':    hype.Database.DBWRITER | hype.Database.DBCREAT,
13975aa16b4c fix LogSearchIndex open permissions
Tero Marttila <terom@fixme.fi>
parents: 66
diff changeset
    67
            'a':    hype.Database.DBWRITER,
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
    68
            'c':    hype.Database.DBWRITER | hype.Database.DBCREAT,
67
13975aa16b4c fix LogSearchIndex open permissions
Tero Marttila <terom@fixme.fi>
parents: 66
diff changeset
    69
            '*':    hype.Database.DBWRITER | hype.Database.DBCREAT | hype.Database.DBTRUNC,
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    70
        }
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    71
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    72
        # look up flags
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    73
        flags = mode_to_flag[mode]
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    74
        
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    75
        # make instance
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    76
        self.db = hype.Database()
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    77
        
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    78
        # open
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    79
        if not self.db.open(path, flags) :
65
8b50694f841e improve search further
Tero Marttila <terom@fixme.fi>
parents: 64
diff changeset
    80
            raise Exception("Index open failed: %s, mode=%s, flags=%#06x: %s" % (path, mode, flags, self.db.err_msg(self.db.error())))
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    81
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    82
    def insert (self, channel, lines) :
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    83
        """
68
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
    84
            Adds a sequence of LogLines from the given LogChannel to the index, and return the number of added items
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    85
        """
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    86
        
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    87
        # validate the LogChannel
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    88
        assert channel.name
68
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
    89
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
    90
        count = 0
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    91
        
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    92
        # iterate
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    93
        for line in lines :
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    94
            # validate the LogLine
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    95
            assert line.offset
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    96
            assert line.timestamp
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    97
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    98
            # create new document
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
    99
            doc = hype.Document()
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   100
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   101
            # line date
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   102
            date = line.timestamp.date()
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   103
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   104
            # convert to UTC timestamp
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   105
            utc_timestamp = calendar.timegm(line.timestamp.utctimetuple())
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   106
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   107
            # ensure that it's not 1900
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   108
            assert date.year != 1900
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   109
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   110
            # add URI
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   111
            doc.add_attr('@uri',        "%s/%s/%d" % (channel.id, date.strftime('%Y-%m-%d'), line.offset))
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   112
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   113
            # add channel id
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   114
            doc.add_attr('channel',     channel.id)
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   115
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   116
            # add type
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   117
            doc.add_attr('type',        str(line.type))
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   118
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   119
            # add UTC timestamp
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   120
            doc.add_attr('timestamp',   str(utc_timestamp))
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   121
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   122
            # add source attribute?
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   123
            if line.source :
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   124
                source_nickname, source_username, source_hostname, source_chanflags = line.source
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   125
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   126
                # XXX: handle source_nickname is None
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   127
                if not source_nickname is None :
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   128
                    source_nickname = str(source_nickname)
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   129
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   130
                doc.add_attr('source_nickname', source_nickname)
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   131
            
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   132
            # add data
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   133
            if line.data :
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   134
                doc.add_text(line.data.encode('utf8'))
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   135
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   136
            # put, "clean up dispensable regions of the overwritten document"
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   137
            if not self.db.put_doc(doc, hype.Database.PDCLEAN) :
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   138
                raise Exeception("Index put_doc failed")
68
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
   139
            
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
   140
            # count
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
   141
            count += 1
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
   142
        
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
   143
        # return
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
   144
        return count
8157c41b3236 improve search form & script
Tero Marttila <terom@fixme.fi>
parents: 67
diff changeset
   145
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   146
    def search_cond (self, cond) :
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   147
        """
74
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
   148
            Search using a raw hype.Condition. Raises NoResultsFound if there aren't any results
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   149
        """
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   150
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   151
        # execute search, unused 'flags' arg stays zero
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   152
        results = self.db.search(cond, 0)
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   153
74
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
   154
        # no results?
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
   155
        if not results :
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
   156
            raise NoResultsFound()
1ab95857d584 handle the 'no search results' case
Tero Marttila <terom@fixme.fi>
parents: 68
diff changeset
   157
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   158
        # iterate over the document IDs
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   159
        for doc_id in results :
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   160
            # load document, this throws an exception...
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   161
            # option constants are hype.Database.GDNOATTR/GDNOTEXT
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   162
            doc = self.db.get_doc(doc_id, 0)
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   163
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   164
            # load the attributes/text
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   165
            channel         = self.channels.lookup(doc.attr('channel'))
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   166
            type            = int(doc.attr('type'))
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   167
            timestamp       = datetime.datetime.fromtimestamp(int(doc.attr('timestamp')), pytz.utc)
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   168
            source_nickname = doc.attr('source_nickname')
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   169
            message         = doc.cat_texts().decode('utf8')
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   170
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   171
            # build+yield to as LogLine
87
39915772f090 update LogSearchIndex to use new LogLine fields
Tero Marttila <terom@fixme.fi>
parents: 74
diff changeset
   172
            yield log_line.LogLine(channel, None, type, timestamp, (source_nickname, None, None, None), None, message)
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   173
    
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   174
    def search (self, options=None, channel=None, phrase=None, order=None, max=None, skip=None) :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   175
        """
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   176
            Search with flexible parameters
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   177
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   178
                options     - bitmask of hype.Condition.*
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   179
                channel     - LogChannel object
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   180
                phrase      - the search query phrase
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   181
                order       - order attribute expression
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   182
                max         - number of results to return
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   183
                skip        - number of results to skip
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   184
        """
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   185
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   186
        # build condition
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   187
        cond = hype.Condition()
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   188
        
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   189
        if options :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   190
            # set options
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   191
            cond.set_options(options)
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   192
        
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   193
        if channel :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   194
            # add channel attribute
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   195
            cond.add_attr("@channel STREQ %s" % (channel.id, ))
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   196
        
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   197
        if phrase :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   198
            # add phrase
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   199
            cond.set_phrase(phrase)
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   200
        
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   201
        if order :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   202
            # set order
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   203
            cond.set_order(order)
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   204
        
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   205
        if max :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   206
            # set max
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   207
            cond.set_max(max)
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   208
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   209
        if skip :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   210
            # set skip
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   211
            cond.set_skip(skip)
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   212
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   213
        # execute
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   214
        return self.search_cond(cond)
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   215
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   216
    def search_simple (self, channel, query, count=None, offset=None) :
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   217
        """
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   218
            Search for lines from the given channel for the given simple query
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   219
        """
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   220
        
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   221
        # use search(), backwards
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   222
        results = list(self.search(
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   223
            # simplified phrase
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   224
            options     = hype.Condition.SIMPLE,
64
cdb6403c2498 beginnings of a LogSearchIndex class
Tero Marttila <terom@fixme.fi>
parents:
diff changeset
   225
66
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   226
            # specific channel
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   227
            channel     = channel,
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   228
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   229
            # given phrase
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   230
            phrase      = query,
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   231
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   232
            # order by timestamp
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   233
            order       = "@timestamp NUMD",
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   234
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   235
            # count/offset
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   236
            max         = count,
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   237
            skip        = offset,
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   238
        ))
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   239
        
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   240
        # reverse
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   241
        return reversed(results)
090ed78ec8fa add count/skip to search results, requires modifications to the swig bindings for HyperEstraier...
Tero Marttila <terom@fixme.fi>
parents: 65
diff changeset
   242