diff options
| author |   Corentin Chary <corentincj@iksaif.net> | 2011-09-21 10:09:50 +0200 |
|---|---|---|
| committer | Corentin Chary <corentincj@iksaif.net> | 2011-09-21 10:09:50 +0200 |
| commit | 14971584af4122cc65352bbf09c8c7b609457a67 (patch) | |
| tree | 186df7e27f1b9de306f9141941127c7a3fb9dd02 /TODO | |
| parent | euscan: blacklist art.gnome.org (diff) | |
| download | euscan-14971584af4122cc65352bbf09c8c7b609457a67.tar.gz euscan-14971584af4122cc65352bbf09c8c7b609457a67.tar.bz2 euscan-14971584af4122cc65352bbf09c8c7b609457a67.zip | |
euscan: robots.txt, timeout, user-agent, ...
- Add a blacklist for robots.txt, we *want* to scan sourceforge
- Set a user-agent that doesn't looks like a browser
- Handle timeouts more carefully
- If brute force detect too much versions, avoid infinite loops
- Handle redirections more carefully
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
Diffstat (limited to 'TODO')
| -rw-r--r-- | TODO | 9 |
1 files changed, 4 insertions, 5 deletions
@@ -4,16 +4,12 @@ TODO euscan ------ -- respect robots.txt (portscout) - check other distros (youri) -- clean blacklist system -- add a way to blacklist versions using standard package tokens - - =x11-drivers/xf86-video-intel-2.14.90* - - >=x11-base/xorg-server-1.10.900 Site Handlers ------------- +- sourceforge: http://sourceforge.net/api/file/index/project-name/vboxgtk/mtime/desc/limit/20/rss http://sourceforge.net/api/release/index/project-id/264534/rss - ftp.kde.org: doesn't scan the "unstable" tree - mysql: should use http://downloads.mysql.com/archives/ - mariadb: should use http://downloads.askmonty.org/MariaDB/+releases/ @@ -22,3 +18,6 @@ euscanwww --------- - add progress options for each command +- add last scan in the footer +- add json/xml for each page +- rss scan world + post ? |