#Change11 :: MOOC Comment Scraper (update)

To recap, I’ve been developing a MOOC Comment Scraper that brings together brief summarised versions of recent blog posts with their comments (‘A ‘Comment Scraper’ for Aggregating Blog Posts with Comments in a MOOC‘). The idea was to provide a quick impression of current MOOC activity but in principle any online event where discussion is distributed over participant blogs could be treated in a similar way. I’ve found the Scraper helpful myself but we all work in different ways so I would like to know how useful such a tool might (or might not) be to other people? Some aspects of design, choice of colours etc, are easily changed. Others that depend on the properties of input feeds, less so. Further development could proceed in several different directions (eg as a research tool) – suggestions are welcome!

Scraping a MOOC for Comments (Based on ‘la vaca de los sinvaca’ – by José Bogado)

As an experiment I will attempt to scrape the Change11 MOOC on a daily basis until its completion. The Scraper output will be published here: MOOC Comment Scraper. An RSS feed (tested OK for Google Reader) is provided by this link: Scraper RSS Feed. I’ve adopted two forms for publication – ‘full’ and ‘abbreviated’ (examples below with links disabled). The full form includes the first line of the latest comments (pingbacks excluded) whereas the abbreviated form omits comment text but retains dates and commenter user names. The full form could be considered to be a derivative work so I’m using this form only with the explicit permission of the blogger or if the blog bears a CC licence permitting derivative works. More full form permissions (currently Jaap, John and Brainysmurf – thanks!) would be very welcome but equally, any request by a blog author not to scrape their blog in any form will be respected. Full Form: 20 Apr: ‘Scraping Comments off MOOCs – good or evil?’ by gbl55 I’ve decided that comment scraping in the full form could be considered a derivative work and so… 20 Apr: What nonsense! Who cares? Publish and be damned! Everyone else does and wh…(AlterEgo) 21 Apr: Thanks AlterEgo but some MOOCers could be offended by unintended juxtaposition of th..(gbl55) 22 Apr: They might be just as offended by the so-called abbreviated form! Get a lawyer or…(AlterEgo) 22 Apr: I just thought I should draw the line there AlterEgo – no I don’t have any legal training bu…(gbl55) 23 Apr: In my learned opinion and notwithstanding the above, hereupon the party of the fir…(LegalEagle) 30 Apr: Thanks LegalEagle for information on copyright violation and risk of my extradition to the…(gbl55) Abbreviated Form: 20 Apr: ‘Scraping Comments off MOOCs – good or evil?’ by gbl55 I’ve decided that comment scraping in the full form could be considered a derivative work and so… Comments by: AlterEgo(20 Apr), gbl55(21 Apr), AlterEgo(22 Apr), gbl55(22 Apr), LegalEagle(23 Apr), gbl55(30 Apr) Notes on the current implementation: I’m no expert on RSS or in coding – some of the following may be misinformed! Current operation is experimental – apologies if comments are altered in peculiar ways. Output is derived only from WordPress and Blogger post and comment feeds. Together these account for a large proportion of MOOC blogs. I have yet to look at other types of feed. Output is limited to the contents of current post and comment feeds and placed in order of date of posting – latest posts first. The Scraper does not aggregate posts and comments (like a reader does) – only the updates provided by feeds are made available at any one time. The Scraper ignores pingbacks. A flurry of very recent pingbacks can push slightly earlier comments out of a current feed. The Scraper is not smart enough to deal with very complex HTML at the beginning of a post or comment and may even default by deleting text. Messages sent by the Scraper itself are indicated by: { … }:  eg {image} if an image is found. A time limit has been set so that only postings less than about 2 months old are included. The greater the number of active blogs the tighter the limit required to avoid generating reams of output.