Currently I’m building a small plugin that’d work with hashtags instead of users, so that tweets with a certain phrase can be harvested and later on displayed, I’m using the tweetnest database and pages for the sake of simplicity, however the crawler its self will be written in java for efficiency, I’m building on top of the tweet_from_table application, since it includes the most up to date classes, I Ordered my netbeans to copy the project, once i had a copy i proceeded to the modification.
I’m thinking along the following lines:
- There will be a table, holding all the hashtags being searched for, this table will include each hashtag and its count, an additional column in the tn_tweets will be added that will include each tweet and which hashtag(s) does it relate to, the challenge here is breaking down the many to many relationship without breaking the GUI.
- The crawler will fire up, query these hashtags, and the last tweet in db for each, and then proceeds searching for them one by one, the result set will be inserted into the database, using the same convention used by the original tweetnest php crawler.
- Finally the results will be displayed using the same GUI used by tweetnest.