]> git.mxchange.org Git - fba.git/log
fba.git
11 months agoContinued:
Roland Häder [Fri, 8 Dec 2023 22:28:48 +0000 (23:28 +0100)]
Continued:
- also search for instances with detected software

11 months agoContinued:
Roland Häder [Fri, 8 Dec 2023 04:44:15 +0000 (05:44 +0100)]
Continued:
- more checks against blacklist

11 months agoContinued:
Roland Häder [Fri, 8 Dec 2023 04:02:57 +0000 (05:02 +0100)]
Continued:
- ops, column and title confused

11 months agoContinued:
Roland Häder [Fri, 8 Dec 2023 03:56:26 +0000 (04:56 +0100)]
Continued:
- added list/scoreboard mode for column 'original_software'

11 months agoContinued:
Roland Häder [Fri, 8 Dec 2023 03:40:46 +0000 (04:40 +0100)]
Continued:
- added column original_software, it contains the original software name, but
  without any version number or " powered by " in it

11 months agoContinued:
Roland Häder [Tue, 5 Dec 2023 01:38:15 +0000 (02:38 +0100)]
Continued:
- more checks against blacklist

11 months agoContinued:
Roland Häder [Tue, 5 Dec 2023 01:01:06 +0000 (02:01 +0100)]
Continued:
- added another alias 'lycheebridge' for 'misskey'

11 months agoContinued:
Roland Häder [Sun, 3 Dec 2023 17:50:00 +0000 (18:50 +0100)]
Continued:
- added alias 'catodon' for 'misskey'

11 months agoContinued:
Roland Häder [Sat, 2 Dec 2023 14:48:22 +0000 (15:48 +0100)]
Continued:
- check blacklist and skip below code
- also handle "error" (int) and "msg" (string) JSON response

11 months agoContinued:
Roland Häder [Fri, 1 Dec 2023 18:52:53 +0000 (19:52 +0100)]
Continued:
- check row[software] if not None, then if it is a relay

11 months agoContinued:
Roland Häder [Thu, 30 Nov 2023 17:54:05 +0000 (18:54 +0100)]
Continued:
- software.is_relay() doesn't allow None/empty strings, let's handle them, too

11 months agoContinued:
Roland Häder [Thu, 30 Nov 2023 04:51:19 +0000 (05:51 +0100)]
Continued:
- daemon.py is already executable

11 months agoContinued:
Roland Häder [Thu, 30 Nov 2023 04:21:36 +0000 (05:21 +0100)]
Continued:
- skip blacklisted domains (aka. instances)

11 months agoContinued:
Roland Häder [Thu, 30 Nov 2023 03:03:02 +0000 (04:03 +0100)]
Continued:
- `foo is None or foo == ""` can be simplified to `foo in [None, ""]`
- added more blacklist check
- improved/added some debug messages

11 months agoContinued:
Roland Häder [Thu, 30 Nov 2023 00:18:08 +0000 (01:18 +0100)]
Continued:
- fixed copy-paste mistakes in debug messages

11 months agoContinued:
Roland Häder [Wed, 29 Nov 2023 22:35:20 +0000 (23:35 +0100)]
Continued:
- ops, one parameter 'key' to much ...

11 months agoContinued:
Roland Häder [Wed, 29 Nov 2023 22:08:45 +0000 (23:08 +0100)]
Continued:
- check domain more again blacklist to avoid bad function invocations
- improved some debug logging messages

11 months agoContinued:
Roland Häder [Wed, 29 Nov 2023 15:16:57 +0000 (16:16 +0100)]
Continued:
- also log 'path' here

11 months agoContinued:
Roland Häder [Wed, 29 Nov 2023 15:02:46 +0000 (16:02 +0100)]
Continued:
- nodeinfo_url is not always a path only, better don't handle it over
- check if parameter 'path' starts with a slash if not 'None'

11 months agoContinued:
Roland Häder [Wed, 29 Nov 2023 00:02:18 +0000 (01:02 +0100)]
Continued:
- should be always aliased ;-)

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 23:17:55 +0000 (00:17 +0100)]
Continued:
- renamed $DOMAINS to $DOMAIN_LIST (Bash script)
- need to exclude None as possible value for parameter 'path'
- added another parked domain

11 months agoBlacklist updated:
Roland Häder [Tue, 28 Nov 2023 22:37:12 +0000 (23:37 +0100)]
Blacklist updated:
- added mass sub-domain flooder
- added fb.me (Facebook websites)

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 22:33:33 +0000 (23:33 +0100)]
Continued:
- these parked domains are the pest of the Internet :-( They just don't contain
  any real content and flood crawl indexes and tables such as 'instances' in
  this software

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 22:04:46 +0000 (23:04 +0100)]
Continued:
- Parameter 'path' need to start with a slash
- checked more against blacklist
- logger message after API returned JSON made similar

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 16:12:38 +0000 (17:12 +0100)]
Continued:
- cached function software.alias()
- moved cached initialization to file header

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 15:03:52 +0000 (16:03 +0100)]
Continued:
- also cache invocations of raise_on() which is being very often invoked

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 14:50:35 +0000 (15:50 +0100)]
Continued:
- first check parameter (better performance)
- cache is_wanted() and is_in_url() invocations

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 07:37:28 +0000 (08:37 +0100)]
Continued:
- blocked more parked domains

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 05:54:55 +0000 (06:54 +0100)]
Continued:
- parked domains blocked

11 months agoContinued:
Roland Häder [Tue, 28 Nov 2023 00:40:23 +0000 (01:40 +0100)]
Continued:
- also skip empty 'href' values
- include 'infos' array, too

11 months agoContinued:
Roland Häder [Mon, 27 Nov 2023 21:47:55 +0000 (22:47 +0100)]
Continued:
- rewrote so all parameters for command fetch_blocks() can have parameter --force
- added parameter --only-none

11 months agoContinued:
Roland Häder [Sun, 26 Nov 2023 13:30:14 +0000 (14:30 +0100)]
Continued:
- added 'sutty.nl' for flooding fediverse with useless "instances"
- it is a mass website hoster

11 months agoContinued:
Roland Häder [Sat, 25 Nov 2023 23:45:04 +0000 (00:45 +0100)]
Continued:
- improved/added debug lines

11 months agoContinued:
Roland Häder [Sat, 25 Nov 2023 13:47:42 +0000 (14:47 +0100)]
Continued:
- added another array dimension where an error message might be set

12 months agoContinued:
Roland Häder [Thu, 23 Nov 2023 01:27:30 +0000 (02:27 +0100)]
Continued:
- check config key against "true"
- improved logger messages
- added some

12 months agoFixed:
Roland Häder [Thu, 23 Nov 2023 00:36:11 +0000 (01:36 +0100)]
Fixed:
- identing must be always aligned

12 months agoContinued:
Roland Häder [Wed, 22 Nov 2023 22:30:42 +0000 (23:30 +0100)]
Continued:
- you can now optionally allow I2P domains being crawled (default: forbidden =
  clear-net)

12 months agoContinued:
Roland Häder [Wed, 22 Nov 2023 22:04:51 +0000 (23:04 +0100)]
Continued:
- improved/added some debug lines

12 months agoContinued:
Roland Häder [Wed, 22 Nov 2023 22:04:25 +0000 (23:04 +0100)]
Continued:
- added dynamic IP address and hostname provider to blacklist

12 months agoContinued:
Roland Häder [Tue, 21 Nov 2023 23:25:07 +0000 (00:25 +0100)]
Continued:
- ops, fixed syntax error

12 months agoContinued:
Roland Häder [Tue, 21 Nov 2023 21:15:08 +0000 (22:15 +0100)]
Continued:
- added parked domain
- added another list for seirdy.one

12 months agoContinued:
Roland Häder [Tue, 21 Nov 2023 18:52:38 +0000 (19:52 +0100)]
Continued:
- also track response time during raised exceptions

12 months agoContinued:
Roland Häder [Tue, 21 Nov 2023 16:23:41 +0000 (17:23 +0100)]
Continued:
- added last_response_time to templates
- moved cookie clearing to proper place (?)

12 months agoContinued:
Roland Häder [Tue, 21 Nov 2023 04:16:45 +0000 (05:16 +0100)]
Continued:
- also need to reduce depth here

12 months agoContinued:
Roland Häder [Mon, 20 Nov 2023 07:56:20 +0000 (08:56 +0100)]
Continued:
- added alias 'write.as' for 'writefreely'

12 months agoContinued:
Roland Häder [Mon, 20 Nov 2023 07:42:02 +0000 (08:42 +0100)]
Continued:
- handle empty peer list differently than fetching none
- added alias 'mbin' for software 'kbin'

12 months agoContinued:
Roland Häder [Mon, 20 Nov 2023 04:27:56 +0000 (05:27 +0100)]
Continued:
- 'misskey' is also a software using obfuscation of blocked instances
- let us better combine None/empty string

12 months agoContinued:
Roland Häder [Mon, 20 Nov 2023 03:47:03 +0000 (04:47 +0100)]
Continued:
- lemmy may also use obfuscation
- also check entries when has_obfuscation is NULL

12 months agoContinued:
Roland Häder [Mon, 20 Nov 2023 02:10:15 +0000 (03:10 +0100)]
Continued:
- more redundant checks against blacklist to avoid bad function invocations

12 months agoContinued:
Roland Häder [Mon, 20 Nov 2023 00:59:26 +0000 (01:59 +0100)]
Continued:
- more blacklist checks
- some formatting

12 months agoContinued:
Roland Häder [Mon, 20 Nov 2023 00:45:15 +0000 (01:45 +0100)]
Continued:
- more named parameters
- added some debug/warning lines

12 months agoContinued:
Roland Häder [Sun, 19 Nov 2023 23:58:49 +0000 (00:58 +0100)]
Continued:
- moved translation list out of function (would reinitialized each time)

12 months agoContinued:
Roland Häder [Sun, 19 Nov 2023 23:40:09 +0000 (00:40 +0100)]
Continued:
- re-indented

12 months agoContinued:
Roland Häder [Sun, 19 Nov 2023 23:28:01 +0000 (00:28 +0100)]
Continued:
- named parameters
- skip None type

12 months agoContinued:
Roland Häder [Sun, 19 Nov 2023 23:06:40 +0000 (00:06 +0100)]
Continued:
- empty strings are not accepted as block reasons, better 'None' instead

12 months agoContinued:
Roland Häder [Sun, 19 Nov 2023 16:38:45 +0000 (17:38 +0100)]
Continued:
- added command "fetch_relaylist" to fetch relays from relaylist.com

12 months agoContinued:
Roland Häder [Sun, 19 Nov 2023 15:43:21 +0000 (16:43 +0100)]
Continued:
- added debug lines (each loop should start with a debug line)

12 months agoContinued:
Roland Häder [Sat, 18 Nov 2023 22:50:03 +0000 (23:50 +0100)]
Continued:
- added another testing site 'tunnel.silicon.moe'

12 months agoContinued:
Roland Häder [Sat, 18 Nov 2023 15:42:33 +0000 (16:42 +0100)]
Continued:
- fixed some errors (ops)

12 months agoContinued:
Roland Häder [Sat, 18 Nov 2023 14:25:47 +0000 (15:25 +0100)]
Continued:
- lesser code block indenting

12 months agoContinued:
Roland Häder [Thu, 16 Nov 2023 12:40:10 +0000 (13:40 +0100)]
Continued:
- duplicate line removed

12 months agoContinued:
Roland Häder [Thu, 9 Nov 2023 14:07:34 +0000 (15:07 +0100)]
Continued:
- not less websites set their (sub-)domain name as software
- maybe let's double-check them if they have fixed it

12 months agoContinued:
Roland Häder [Mon, 6 Nov 2023 10:53:32 +0000 (11:53 +0100)]
Continued:
- added another alias for misskey

12 months agoContinued:
Roland Häder [Mon, 6 Nov 2023 03:02:06 +0000 (04:02 +0100)]
Continued:
- moved url list to fba/helpers/blocklists.py
- always update count of obfuscated domains

12 months agoContinued:
Roland Häder [Thu, 2 Nov 2023 04:35:25 +0000 (05:35 +0100)]
Continued:
- added missing parameter for SQL queries
- added some aliases for misskey

12 months agoContinued:
Roland Häder [Thu, 2 Nov 2023 00:17:52 +0000 (01:17 +0100)]
Continued:
- maybe also -rc came here

12 months agoContinued:
Roland Häder [Thu, 26 Oct 2023 05:49:14 +0000 (07:49 +0200)]
Continued:
- this instance has repeatedly (3x) caused an OOM (Out-Of-Memory) error here

13 months agoContinued:
Roland Häder [Tue, 24 Oct 2023 18:45:11 +0000 (20:45 +0200)]
Continued:
- some Friendica instances may have no table tag

13 months agoContinued:
Roland Häder [Sun, 22 Oct 2023 12:59:19 +0000 (14:59 +0200)]
Continued:
- wrong place to alias, this way all *key software became misskey and no other
  would be checked

13 months agoContinued:
Roland Häder [Sun, 22 Oct 2023 04:28:48 +0000 (06:28 +0200)]
Continued:
- skip empty domain names before is_wanted() is invocated

13 months agoContinued:
Roland Häder [Fri, 20 Oct 2023 04:58:47 +0000 (06:58 +0200)]
Continued:
- software doesn't need to be aliases each round as the variable isn't assigned
  inside the loop
- added check in software_helper.alias() if parameter 'software' is an empty
  string
- don't attempt to alias empty software string

13 months agoContinued:
Roland Häder [Wed, 18 Oct 2023 21:27:43 +0000 (23:27 +0200)]
Continued:
- more cases of empty strings

13 months agoContinued:
Roland Häder [Wed, 18 Oct 2023 17:06:59 +0000 (19:06 +0200)]
Continued:
- tpzo fixed

13 months agoContinued:
Roland Häder [Sat, 14 Oct 2023 00:55:45 +0000 (02:55 +0200)]
Continued:
- also ignore empty content here, too

13 months agoContinued:
Roland Häder [Fri, 13 Oct 2023 00:31:50 +0000 (02:31 +0200)]
Continued:
- added view for recently added instances

13 months agoContinued:
Roland Häder [Thu, 12 Oct 2023 23:17:07 +0000 (01:17 +0200)]
Continued:
- added parameter --no-detection for rechecking instances with no detection mode
  being set which might happen when the server is down or the used software was
  nowhere stated

13 months agoContinued:
Roland Häder [Thu, 12 Oct 2023 23:09:24 +0000 (01:09 +0200)]
Continued:
- no need for these checks anymore

13 months agoContinued:
Roland Häder [Thu, 12 Oct 2023 22:53:20 +0000 (00:53 +0200)]
Continued:
- cut of any port part from `components.netloc`

13 months agoContinued:
Roland Häder [Wed, 11 Oct 2023 17:53:07 +0000 (19:53 +0200)]
Continued:
- no longer needed as INFO messages because below code also has logged a similar
  message on INFO level

13 months agoContinued:
Roland Häder [Wed, 11 Oct 2023 11:42:32 +0000 (13:42 +0200)]
Continued:
- also need to skip empty strings

13 months agoContinued:
Roland Häder [Sat, 7 Oct 2023 12:42:50 +0000 (14:42 +0200)]
Continued:
- moved fba/deprecated.py to deprecated/commands.py
- fixed many issues pylint has reported

13 months agoContinued:
Roland Häder [Sat, 7 Oct 2023 11:03:59 +0000 (13:03 +0200)]
Continued:
- each kid on this world seem to have their own misskey clone now only by
  replacing the loading graphic?

13 months agoContinued:
Roland Häder [Fri, 6 Oct 2023 13:59:34 +0000 (15:59 +0200)]
Continued:
- added some debug messages
- formatted command
- added alias 'hajkey' as 'misskey' alias

13 months agoContinued:
Roland Häder [Thu, 5 Oct 2023 00:29:07 +0000 (02:29 +0200)]
Continued:
- don't attempt to fetch peers from fetch_instances() function when the software
  is a relay, command fetch_relay() should be used instead to fetch relay's
  peer list

13 months agoContinued:
Roland Häder [Wed, 4 Oct 2023 08:18:54 +0000 (10:18 +0200)]
Continued:
- some people made a website which redirected to other redirect-only domains,
  e.g. start with 'social.golangengine.de' and you see what I mean
- if fetch_instances() is used here, it will cause a permanent redirect and
  later a stackoverflow

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 19:04:09 +0000 (21:04 +0200)]
Continued:
- local variable 'block' was defined only within the loop, not outside of it
- I actually have to use local variable 'blocker' instead

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 13:32:29 +0000 (15:32 +0200)]
Continued:
- also set has_obfuscation here, too

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 13:24:57 +0000 (15:24 +0200)]
Continued:
- change has_obfuscation when detected, not when a domain wasn't deobfuscated
- also count de-/obfuscated domains

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 09:09:07 +0000 (11:09 +0200)]
Continued:
- some people entered invalid domains names with two dots (foo..bar)

13 months agoContinued:
Roland Häder [Mon, 2 Oct 2023 07:56:43 +0000 (09:56 +0200)]
Continued:
- ordered SELECT statement for update_nodeinfo() command
- added --no-auto as another "filter" parameter
- don't check domains that turn e.g. into an IP address before redirecting to
  them

13 months agoContinued:
Roland Häder [Sat, 30 Sep 2023 21:54:51 +0000 (23:54 +0200)]
Continued:
- added some more checks to avoid exceptions

13 months agoContinued:
Roland Häder [Sat, 30 Sep 2023 11:29:26 +0000 (13:29 +0200)]
Continued:
- row["software"] is the proper way here, ops

13 months agoContinued:
Roland Häder [Sat, 30 Sep 2023 11:11:41 +0000 (13:11 +0200)]
Continued:
- Paula has finally seen the wrong outcome of publishing a #FediBlock list
  publicly:
  "Too many people use blocklists as-is and don't use their own brain. Blindly
   blocking instances because someone else says so is not good."

13 months agoContinued:
Roland Häder [Fri, 29 Sep 2023 17:23:18 +0000 (19:23 +0200)]
Continued:
- added another alias for 'misskey'

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:13:58 +0000 (12:13 +0200)]
Continued:
- commit after each CSV row being handled
- tpzo fixed in module name
- always add domain to domains list

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:10:42 +0000 (12:10 +0200)]
Continued:
- if later more "hidden" blocklists are added --domain=foo can only process
  foo's blocklist

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:08:16 +0000 (12:08 +0200)]
Continued:
- old left-over

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:06:54 +0000 (12:06 +0200)]
Continued:
- some CSV files have their headers upper-case

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:00:19 +0000 (12:00 +0200)]
Continued:
- added command 'fetch_csv' which fetches CSV files and processes them for
  further instance discovery and blocklist expansion
- introduced function processing.csv_block() which does the above processing
- return non-zero exit code when source was queried to recently

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 07:40:21 +0000 (09:40 +0200)]
Continued:
- has_obfuscation should be only an indicator if a single domain is obfuscated
  then this should be set to True, otherwise False