]>
git.mxchange.org Git - fba.git/log
Roland Häder [Thu, 15 Feb 2024 17:58:51 +0000 (18:58 +0100)]
Continued:
- added network type 'vebinet' (Mozilla's Fediverse software) for peer retrival
- domain blocks aren't supported (yet)
Roland Häder [Thu, 15 Feb 2024 13:30:07 +0000 (14:30 +0100)]
Continued:
- first simple variable checks
- next "lighter" functions being invoked
- last "heavy" function, which means possible database queries
Roland Häder [Tue, 13 Feb 2024 18:31:31 +0000 (19:31 +0100)]
Continued:
- added some more plain-text block lists (thanks to @Kromonos)
Roland Häder [Mon, 12 Feb 2024 15:45:46 +0000 (16:45 +0100)]
Continued:
- check parameters first, then "expensive" function invocation's returned value
- command fetch_instances should always fetch instances from a provided domain
name
- only when --software=bar is used and no --force parameter was given
Roland Häder [Thu, 8 Feb 2024 17:00:57 +0000 (18:00 +0100)]
Continued:
- make --domain parameter for command fetch_instances optional so --software
can be handled, too
- skip recently crawled domains in same loop
- args.<domain|software> can be both of type None, too
Roland Häder [Wed, 7 Feb 2024 19:26:24 +0000 (20:26 +0100)]
Continued:
- github isn't any fediverse instance
- https://github.io redirects to a bad URL
Roland Häder [Wed, 7 Feb 2024 07:55:13 +0000 (08:55 +0100)]
Continued:
- it was imported from previous code and seem not to work everywhere
- so let's remove the non-working part
Roland Häder [Sun, 4 Feb 2024 08:29:38 +0000 (09:29 +0100)]
Continued:
- added alias 'rosekey' for 'misskey'
- also 'smithereen' is a federating (at least peer list is provided) software
Roland Häder [Sat, 3 Feb 2024 19:39:39 +0000 (20:39 +0100)]
Continued:
- maybe there is no unsorted list or table at all
Roland Häder [Wed, 24 Jan 2024 17:14:14 +0000 (18:14 +0100)]
Continued:
- 'continue' if an exception is thrown because that instance can be "ignored"
Roland Häder [Wed, 24 Jan 2024 05:18:53 +0000 (06:18 +0100)]
Continued:
- handled unregistered instances
Roland Häder [Tue, 16 Jan 2024 19:18:16 +0000 (20:18 +0100)]
Continued/WIP:
- in commands.fetch_instances() added initialization of variables 'rows'
- also moved fetching rows into if() block
- commented some more code
Roland Häder [Tue, 16 Jan 2024 02:48:35 +0000 (03:48 +0100)]
Continued:
- added alias "kmyblue" for mastodon
Roland Häder [Sun, 14 Jan 2024 01:25:10 +0000 (02:25 +0100)]
WIP:
- rewritten fetch_instances towars allowing also --software=foo as an
alternative parameter
Roland Häder [Fri, 12 Jan 2024 09:00:40 +0000 (10:00 +0100)]
Continued:
- changed string "true/false" to boolean
- please update your configuration file (all occurances!)
Roland Häder [Fri, 12 Jan 2024 03:56:07 +0000 (04:56 +0100)]
Continued:
- need to negate this state: only skip/reject .i2p domains when they are not
allowed by configuration (which is default)
Roland Häder [Wed, 10 Jan 2024 21:52:50 +0000 (22:52 +0100)]
Continued:
- moved utils.fetch_url() to module network as this is network-related
- added some debug lines
Roland Häder [Tue, 2 Jan 2024 19:45:51 +0000 (20:45 +0100)]
Continued:
- need to remove prefix "re:" before cleaning up software, else all software is called "re"
- added alias "lovers" for "misskey"
Roland Häder [Tue, 2 Jan 2024 19:30:01 +0000 (20:30 +0100)]
Continued:
- added alias "miraiskey" for "misskey"
- moved all those aliases to "private" variable _misskey_aliases
Roland Häder [Sat, 23 Dec 2023 16:10:59 +0000 (17:10 +0100)]
Continued:
- fediverse.observer has changed their API to Graph (JSON POST)
- domain singleuser.club blacklisted, this domain started flooding with
sub-domains which have wwXX as another sub-domain
Roland Häder [Fri, 22 Dec 2023 07:44:52 +0000 (08:44 +0100)]
Continued:
- need to cut off everything after hash symbol because that is for JavaScript
click-event loaded content anyway
- prevented a few empty/None strings for invoking tidyup.domain()
- improved a few log messages
Roland Häder [Sat, 16 Dec 2023 13:19:57 +0000 (14:19 +0100)]
Continued:
- don't allow empty peer list in federation.add_peers()
Roland Häder [Sat, 16 Dec 2023 08:51:12 +0000 (09:51 +0100)]
Continued:
- added missing 'continue'
- added some debug messages
- skip empty/NoneType domain names
Roland Häder [Tue, 12 Dec 2023 06:07:23 +0000 (07:07 +0100)]
Continued:
- maybe error[json][error] is not a dict
Roland Häder [Tue, 12 Dec 2023 02:34:16 +0000 (03:34 +0100)]
Continued:
- added "error_message" JSON error message
Roland Häder [Mon, 11 Dec 2023 06:57:39 +0000 (07:57 +0100)]
Continued:
- also check blacklist first, before invoking instances.is_recent()
Roland Häder [Sun, 10 Dec 2023 23:54:01 +0000 (00:54 +0100)]
Continued:
- also check blacklist in instances.is_recent() function
- first check against blacklist, then if it is recent (in commands.py)
Roland Häder [Sun, 10 Dec 2023 22:44:35 +0000 (23:44 +0100)]
Continued:
- added alias 'goblin' for 'misskey'
- added debug line
Roland Häder [Sun, 10 Dec 2023 22:08:50 +0000 (23:08 +0100)]
Continued:
- Python still needs a backslash here
Roland Häder [Sun, 10 Dec 2023 08:04:23 +0000 (09:04 +0100)]
Continued:
- check validity of href URL before parsing it (controlled skip instead of
uncontrolled raised exception)
Roland Häder [Sun, 10 Dec 2023 07:20:07 +0000 (08:20 +0100)]
Continued:
- reformatted
Roland Häder [Sun, 10 Dec 2023 05:15:09 +0000 (06:15 +0100)]
Continued:
- some relays may have removed/never set nodeinfo URL, these need to be skipped
to avoid exception in fetch_url() function
Roland Häder [Sun, 10 Dec 2023 04:50:06 +0000 (05:50 +0100)]
Continued:
- command 'fetch_txt' supports now --force parameter
- check against blacklist (wasn't in is_registered())
- some debug messages added
Roland Häder [Sun, 10 Dec 2023 04:06:26 +0000 (05:06 +0100)]
Continued:
- formatted SQL statements
Roland Häder [Sun, 10 Dec 2023 02:42:22 +0000 (03:42 +0100)]
Continued:
- forgot to vacuum-clean it
Roland Häder [Sun, 10 Dec 2023 01:06:59 +0000 (02:06 +0100)]
Continued:
- simplified code, no need for this extra indenting
Roland Häder [Sat, 9 Dec 2023 07:15:42 +0000 (08:15 +0100)]
Continued:
- combined None and "" (empty string) for lesser code
- added/improved debug messages
- got rid of one local variable, added another one ;-)
Roland Häder [Sat, 9 Dec 2023 07:03:01 +0000 (08:03 +0100)]
Continued:
- improved logger messages
Roland Häder [Sat, 9 Dec 2023 05:14:48 +0000 (06:14 +0100)]
Continued:
- logged domain name
- better check variable software against None and empty string
Roland Häder [Sat, 9 Dec 2023 05:04:20 +0000 (06:04 +0100)]
Continued:
- initially set original_software, maybe it needs fixing later
Roland Häder [Sat, 9 Dec 2023 05:01:38 +0000 (06:01 +0100)]
Continued:
- need to catch any network-related exceptions to reduce _DEPTH counter
Roland Häder [Sat, 9 Dec 2023 04:10:38 +0000 (05:10 +0100)]
Continued:
- moved original_software next to 'software' column
- replaced own _cache dict with @lru_cache
Roland Häder [Sat, 9 Dec 2023 03:03:46 +0000 (04:03 +0100)]
Continued:
- check if domain is valid before checking if it is blacklisted, else an
exception will raise
Roland Häder [Sat, 9 Dec 2023 02:46:25 +0000 (03:46 +0100)]
Continued:
- check all URLs against validator
Roland Häder [Fri, 8 Dec 2023 22:28:48 +0000 (23:28 +0100)]
Continued:
- also search for instances with detected software
Roland Häder [Fri, 8 Dec 2023 04:44:15 +0000 (05:44 +0100)]
Continued:
- more checks against blacklist
Roland Häder [Fri, 8 Dec 2023 04:02:57 +0000 (05:02 +0100)]
Continued:
- ops, column and title confused
Roland Häder [Fri, 8 Dec 2023 03:56:26 +0000 (04:56 +0100)]
Continued:
- added list/scoreboard mode for column 'original_software'
Roland Häder [Fri, 8 Dec 2023 03:40:46 +0000 (04:40 +0100)]
Continued:
- added column original_software, it contains the original software name, but
without any version number or " powered by " in it
Roland Häder [Tue, 5 Dec 2023 01:38:15 +0000 (02:38 +0100)]
Continued:
- more checks against blacklist
Roland Häder [Tue, 5 Dec 2023 01:01:06 +0000 (02:01 +0100)]
Continued:
- added another alias 'lycheebridge' for 'misskey'
Roland Häder [Sun, 3 Dec 2023 17:50:00 +0000 (18:50 +0100)]
Continued:
- added alias 'catodon' for 'misskey'
Roland Häder [Sat, 2 Dec 2023 14:48:22 +0000 (15:48 +0100)]
Continued:
- check blacklist and skip below code
- also handle "error" (int) and "msg" (string) JSON response
Roland Häder [Fri, 1 Dec 2023 18:52:53 +0000 (19:52 +0100)]
Continued:
- check row[software] if not None, then if it is a relay
Roland Häder [Thu, 30 Nov 2023 17:54:05 +0000 (18:54 +0100)]
Continued:
- software.is_relay() doesn't allow None/empty strings, let's handle them, too
Roland Häder [Thu, 30 Nov 2023 04:51:19 +0000 (05:51 +0100)]
Continued:
- daemon.py is already executable
Roland Häder [Thu, 30 Nov 2023 04:21:36 +0000 (05:21 +0100)]
Continued:
- skip blacklisted domains (aka. instances)
Roland Häder [Thu, 30 Nov 2023 03:03:02 +0000 (04:03 +0100)]
Continued:
- `foo is None or foo == ""` can be simplified to `foo in [None, ""]`
- added more blacklist check
- improved/added some debug messages
Roland Häder [Thu, 30 Nov 2023 00:18:08 +0000 (01:18 +0100)]
Continued:
- fixed copy-paste mistakes in debug messages
Roland Häder [Wed, 29 Nov 2023 22:35:20 +0000 (23:35 +0100)]
Continued:
- ops, one parameter 'key' to much ...
Roland Häder [Wed, 29 Nov 2023 22:08:45 +0000 (23:08 +0100)]
Continued:
- check domain more again blacklist to avoid bad function invocations
- improved some debug logging messages
Roland Häder [Wed, 29 Nov 2023 15:16:57 +0000 (16:16 +0100)]
Continued:
- also log 'path' here
Roland Häder [Wed, 29 Nov 2023 15:02:46 +0000 (16:02 +0100)]
Continued:
- nodeinfo_url is not always a path only, better don't handle it over
- check if parameter 'path' starts with a slash if not 'None'
Roland Häder [Wed, 29 Nov 2023 00:02:18 +0000 (01:02 +0100)]
Continued:
- should be always aliased ;-)
Roland Häder [Tue, 28 Nov 2023 23:17:55 +0000 (00:17 +0100)]
Continued:
- renamed $DOMAINS to $DOMAIN_LIST (Bash script)
- need to exclude None as possible value for parameter 'path'
- added another parked domain
Roland Häder [Tue, 28 Nov 2023 22:37:12 +0000 (23:37 +0100)]
Blacklist updated:
- added mass sub-domain flooder
- added fb.me (Facebook websites)
Roland Häder [Tue, 28 Nov 2023 22:33:33 +0000 (23:33 +0100)]
Continued:
- these parked domains are the pest of the Internet :-( They just don't contain
any real content and flood crawl indexes and tables such as 'instances' in
this software
Roland Häder [Tue, 28 Nov 2023 22:04:46 +0000 (23:04 +0100)]
Continued:
- Parameter 'path' need to start with a slash
- checked more against blacklist
- logger message after API returned JSON made similar
Roland Häder [Tue, 28 Nov 2023 16:12:38 +0000 (17:12 +0100)]
Continued:
- cached function software.alias()
- moved cached initialization to file header
Roland Häder [Tue, 28 Nov 2023 15:03:52 +0000 (16:03 +0100)]
Continued:
- also cache invocations of raise_on() which is being very often invoked
Roland Häder [Tue, 28 Nov 2023 14:50:35 +0000 (15:50 +0100)]
Continued:
- first check parameter (better performance)
- cache is_wanted() and is_in_url() invocations
Roland Häder [Tue, 28 Nov 2023 07:37:28 +0000 (08:37 +0100)]
Continued:
- blocked more parked domains
Roland Häder [Tue, 28 Nov 2023 05:54:55 +0000 (06:54 +0100)]
Continued:
- parked domains blocked
Roland Häder [Tue, 28 Nov 2023 00:40:23 +0000 (01:40 +0100)]
Continued:
- also skip empty 'href' values
- include 'infos' array, too
Roland Häder [Mon, 27 Nov 2023 21:47:55 +0000 (22:47 +0100)]
Continued:
- rewrote so all parameters for command fetch_blocks() can have parameter --force
- added parameter --only-none
Roland Häder [Sun, 26 Nov 2023 13:30:14 +0000 (14:30 +0100)]
Continued:
- added 'sutty.nl' for flooding fediverse with useless "instances"
- it is a mass website hoster
Roland Häder [Sat, 25 Nov 2023 23:45:04 +0000 (00:45 +0100)]
Continued:
- improved/added debug lines
Roland Häder [Sat, 25 Nov 2023 13:47:42 +0000 (14:47 +0100)]
Continued:
- added another array dimension where an error message might be set
Roland Häder [Thu, 23 Nov 2023 01:27:30 +0000 (02:27 +0100)]
Continued:
- check config key against "true"
- improved logger messages
- added some
Roland Häder [Thu, 23 Nov 2023 00:36:11 +0000 (01:36 +0100)]
Fixed:
- identing must be always aligned
Roland Häder [Wed, 22 Nov 2023 22:30:42 +0000 (23:30 +0100)]
Continued:
- you can now optionally allow I2P domains being crawled (default: forbidden =
clear-net)
Roland Häder [Wed, 22 Nov 2023 22:04:51 +0000 (23:04 +0100)]
Continued:
- improved/added some debug lines
Roland Häder [Wed, 22 Nov 2023 22:04:25 +0000 (23:04 +0100)]
Continued:
- added dynamic IP address and hostname provider to blacklist
Roland Häder [Tue, 21 Nov 2023 23:25:07 +0000 (00:25 +0100)]
Continued:
- ops, fixed syntax error
Roland Häder [Tue, 21 Nov 2023 21:15:08 +0000 (22:15 +0100)]
Continued:
- added parked domain
- added another list for seirdy.one
Roland Häder [Tue, 21 Nov 2023 18:52:38 +0000 (19:52 +0100)]
Continued:
- also track response time during raised exceptions
Roland Häder [Tue, 21 Nov 2023 16:23:41 +0000 (17:23 +0100)]
Continued:
- added last_response_time to templates
- moved cookie clearing to proper place (?)
Roland Häder [Tue, 21 Nov 2023 04:16:45 +0000 (05:16 +0100)]
Continued:
- also need to reduce depth here
Roland Häder [Mon, 20 Nov 2023 07:56:20 +0000 (08:56 +0100)]
Continued:
- added alias 'write.as' for 'writefreely'
Roland Häder [Mon, 20 Nov 2023 07:42:02 +0000 (08:42 +0100)]
Continued:
- handle empty peer list differently than fetching none
- added alias 'mbin' for software 'kbin'
Roland Häder [Mon, 20 Nov 2023 04:27:56 +0000 (05:27 +0100)]
Continued:
- 'misskey' is also a software using obfuscation of blocked instances
- let us better combine None/empty string
Roland Häder [Mon, 20 Nov 2023 03:47:03 +0000 (04:47 +0100)]
Continued:
- lemmy may also use obfuscation
- also check entries when has_obfuscation is NULL
Roland Häder [Mon, 20 Nov 2023 02:10:15 +0000 (03:10 +0100)]
Continued:
- more redundant checks against blacklist to avoid bad function invocations
Roland Häder [Mon, 20 Nov 2023 00:59:26 +0000 (01:59 +0100)]
Continued:
- more blacklist checks
- some formatting
Roland Häder [Mon, 20 Nov 2023 00:45:15 +0000 (01:45 +0100)]
Continued:
- more named parameters
- added some debug/warning lines
Roland Häder [Sun, 19 Nov 2023 23:58:49 +0000 (00:58 +0100)]
Continued:
- moved translation list out of function (would reinitialized each time)
Roland Häder [Sun, 19 Nov 2023 23:40:09 +0000 (00:40 +0100)]
Continued:
- re-indented
Roland Häder [Sun, 19 Nov 2023 23:28:01 +0000 (00:28 +0100)]
Continued:
- named parameters
- skip None type
Roland Häder [Sun, 19 Nov 2023 23:06:40 +0000 (00:06 +0100)]
Continued:
- empty strings are not accepted as block reasons, better 'None' instead
Roland Häder [Sun, 19 Nov 2023 16:38:45 +0000 (17:38 +0100)]
Continued:
- added command "fetch_relaylist" to fetch relays from relaylist.com