]> git.mxchange.org Git - fba.git/log
fba.git
12 months agoContinued:
Roland Häder [Mon, 6 Nov 2023 10:53:32 +0000 (11:53 +0100)]
Continued:
- added another alias for misskey

12 months agoContinued:
Roland Häder [Mon, 6 Nov 2023 03:02:06 +0000 (04:02 +0100)]
Continued:
- moved url list to fba/helpers/blocklists.py
- always update count of obfuscated domains

12 months agoContinued:
Roland Häder [Thu, 2 Nov 2023 04:35:25 +0000 (05:35 +0100)]
Continued:
- added missing parameter for SQL queries
- added some aliases for misskey

12 months agoContinued:
Roland Häder [Thu, 2 Nov 2023 00:17:52 +0000 (01:17 +0100)]
Continued:
- maybe also -rc came here

12 months agoContinued:
Roland Häder [Thu, 26 Oct 2023 05:49:14 +0000 (07:49 +0200)]
Continued:
- this instance has repeatedly (3x) caused an OOM (Out-Of-Memory) error here

12 months agoContinued:
Roland Häder [Tue, 24 Oct 2023 18:45:11 +0000 (20:45 +0200)]
Continued:
- some Friendica instances may have no table tag

12 months agoContinued:
Roland Häder [Sun, 22 Oct 2023 12:59:19 +0000 (14:59 +0200)]
Continued:
- wrong place to alias, this way all *key software became misskey and no other
  would be checked

12 months agoContinued:
Roland Häder [Sun, 22 Oct 2023 04:28:48 +0000 (06:28 +0200)]
Continued:
- skip empty domain names before is_wanted() is invocated

12 months agoContinued:
Roland Häder [Fri, 20 Oct 2023 04:58:47 +0000 (06:58 +0200)]
Continued:
- software doesn't need to be aliases each round as the variable isn't assigned
  inside the loop
- added check in software_helper.alias() if parameter 'software' is an empty
  string
- don't attempt to alias empty software string

12 months agoContinued:
Roland Häder [Wed, 18 Oct 2023 21:27:43 +0000 (23:27 +0200)]
Continued:
- more cases of empty strings

12 months agoContinued:
Roland Häder [Wed, 18 Oct 2023 17:06:59 +0000 (19:06 +0200)]
Continued:
- tpzo fixed

12 months agoContinued:
Roland Häder [Sat, 14 Oct 2023 00:55:45 +0000 (02:55 +0200)]
Continued:
- also ignore empty content here, too

13 months agoContinued:
Roland Häder [Fri, 13 Oct 2023 00:31:50 +0000 (02:31 +0200)]
Continued:
- added view for recently added instances

13 months agoContinued:
Roland Häder [Thu, 12 Oct 2023 23:17:07 +0000 (01:17 +0200)]
Continued:
- added parameter --no-detection for rechecking instances with no detection mode
  being set which might happen when the server is down or the used software was
  nowhere stated

13 months agoContinued:
Roland Häder [Thu, 12 Oct 2023 23:09:24 +0000 (01:09 +0200)]
Continued:
- no need for these checks anymore

13 months agoContinued:
Roland Häder [Thu, 12 Oct 2023 22:53:20 +0000 (00:53 +0200)]
Continued:
- cut of any port part from `components.netloc`

13 months agoContinued:
Roland Häder [Wed, 11 Oct 2023 17:53:07 +0000 (19:53 +0200)]
Continued:
- no longer needed as INFO messages because below code also has logged a similar
  message on INFO level

13 months agoContinued:
Roland Häder [Wed, 11 Oct 2023 11:42:32 +0000 (13:42 +0200)]
Continued:
- also need to skip empty strings

13 months agoContinued:
Roland Häder [Sat, 7 Oct 2023 12:42:50 +0000 (14:42 +0200)]
Continued:
- moved fba/deprecated.py to deprecated/commands.py
- fixed many issues pylint has reported

13 months agoContinued:
Roland Häder [Sat, 7 Oct 2023 11:03:59 +0000 (13:03 +0200)]
Continued:
- each kid on this world seem to have their own misskey clone now only by
  replacing the loading graphic?

13 months agoContinued:
Roland Häder [Fri, 6 Oct 2023 13:59:34 +0000 (15:59 +0200)]
Continued:
- added some debug messages
- formatted command
- added alias 'hajkey' as 'misskey' alias

13 months agoContinued:
Roland Häder [Thu, 5 Oct 2023 00:29:07 +0000 (02:29 +0200)]
Continued:
- don't attempt to fetch peers from fetch_instances() function when the software
  is a relay, command fetch_relay() should be used instead to fetch relay's
  peer list

13 months agoContinued:
Roland Häder [Wed, 4 Oct 2023 08:18:54 +0000 (10:18 +0200)]
Continued:
- some people made a website which redirected to other redirect-only domains,
  e.g. start with 'social.golangengine.de' and you see what I mean
- if fetch_instances() is used here, it will cause a permanent redirect and
  later a stackoverflow

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 19:04:09 +0000 (21:04 +0200)]
Continued:
- local variable 'block' was defined only within the loop, not outside of it
- I actually have to use local variable 'blocker' instead

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 13:32:29 +0000 (15:32 +0200)]
Continued:
- also set has_obfuscation here, too

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 13:24:57 +0000 (15:24 +0200)]
Continued:
- change has_obfuscation when detected, not when a domain wasn't deobfuscated
- also count de-/obfuscated domains

13 months agoContinued:
Roland Häder [Tue, 3 Oct 2023 09:09:07 +0000 (11:09 +0200)]
Continued:
- some people entered invalid domains names with two dots (foo..bar)

13 months agoContinued:
Roland Häder [Mon, 2 Oct 2023 07:56:43 +0000 (09:56 +0200)]
Continued:
- ordered SELECT statement for update_nodeinfo() command
- added --no-auto as another "filter" parameter
- don't check domains that turn e.g. into an IP address before redirecting to
  them

13 months agoContinued:
Roland Häder [Sat, 30 Sep 2023 21:54:51 +0000 (23:54 +0200)]
Continued:
- added some more checks to avoid exceptions

13 months agoContinued:
Roland Häder [Sat, 30 Sep 2023 11:29:26 +0000 (13:29 +0200)]
Continued:
- row["software"] is the proper way here, ops

13 months agoContinued:
Roland Häder [Sat, 30 Sep 2023 11:11:41 +0000 (13:11 +0200)]
Continued:
- Paula has finally seen the wrong outcome of publishing a #FediBlock list
  publicly:
  "Too many people use blocklists as-is and don't use their own brain. Blindly
   blocking instances because someone else says so is not good."

13 months agoContinued:
Roland Häder [Fri, 29 Sep 2023 17:23:18 +0000 (19:23 +0200)]
Continued:
- added another alias for 'misskey'

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:13:58 +0000 (12:13 +0200)]
Continued:
- commit after each CSV row being handled
- tpzo fixed in module name
- always add domain to domains list

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:10:42 +0000 (12:10 +0200)]
Continued:
- if later more "hidden" blocklists are added --domain=foo can only process
  foo's blocklist

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:08:16 +0000 (12:08 +0200)]
Continued:
- old left-over

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:06:54 +0000 (12:06 +0200)]
Continued:
- some CSV files have their headers upper-case

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 10:00:19 +0000 (12:00 +0200)]
Continued:
- added command 'fetch_csv' which fetches CSV files and processes them for
  further instance discovery and blocklist expansion
- introduced function processing.csv_block() which does the above processing
- return non-zero exit code when source was queried to recently

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 07:40:21 +0000 (09:40 +0200)]
Continued:
- has_obfuscation should be only an indicator if a single domain is obfuscated
  then this should be set to True, otherwise False

14 months agoContinued:
Roland Häder [Tue, 12 Sep 2023 01:40:31 +0000 (03:40 +0200)]
Continued:
- please execute SQL command:
  "UPDATE instances SET command = 'redirect_target' WHERE command = 'fetch_generator';"
- yes, it is done during detection mode 'generator' but it was discovered as
  redirection target

14 months agoContinued:
Roland Häder [Mon, 11 Sep 2023 20:27:46 +0000 (22:27 +0200)]
Continued:
- some people think that CamelCase.Domain is something to be proud of
- truth is, that those upper-case characters are being lower-cased before a DNS
  server is been queried

14 months agoContinued:
Roland Häder [Mon, 11 Sep 2023 19:53:11 +0000 (21:53 +0200)]
Continued:
- instances.has_pending() raises an exception if domain is not registered yet
- instances.update() checks is_registered() first, then has_pending()

14 months agoContinued:
Roland Häder [Mon, 11 Sep 2023 18:59:11 +0000 (20:59 +0200)]
Continued:
- some url.netloc may contain port number which needs to be split off

14 months agoContinued:
Roland Häder [Fri, 8 Sep 2023 16:50:17 +0000 (18:50 +0200)]
Continued:
- more checks on empty/None domain names

14 months agoContinued:
Roland Häder [Wed, 6 Sep 2023 08:58:23 +0000 (10:58 +0200)]
Continued:
- added last_response_time which is a float that stores the last response time
- you have to run following SQL statement on your blocks.db:
  ALTER TABLE instances ADD last_response_time FLOAT NULL DEFAULT NULL;

14 months agoContinued:
Roland Häder [Wed, 6 Sep 2023 02:02:26 +0000 (04:02 +0200)]
Continued:
- introduced software_helper.is_relay() function to check if given software name
  is a supported relay software
- federation.fetch_instances() will throw an exception if invoked with relay
  software
- also command fetch_instances avoids them

14 months agoContinued:
Roland Häder [Wed, 6 Sep 2023 01:37:48 +0000 (03:37 +0200)]
Continued:
- fetch_relays now supports --software=foo parameter
- added support for 'pub-relay' relays, they provide their peers over their
  nodeinfo URL (see element metadata -> peers)

14 months agoContinued:
Roland Häder [Wed, 6 Sep 2023 00:43:17 +0000 (02:43 +0200)]
Continued:
- skip empty domain names
- also strip spaces on start/end away

14 months agoContinued:
Roland Häder [Tue, 5 Sep 2023 11:54:35 +0000 (13:54 +0200)]
Continued:
- tpzo fixed

14 months agoContinued:
Roland Häder [Mon, 4 Sep 2023 23:15:53 +0000 (01:15 +0200)]
Continued:
- no need for double-fetching generic JSON API

14 months agoContinued:
Roland Häder [Mon, 4 Sep 2023 22:44:32 +0000 (00:44 +0200)]
Continued:
- skip more empty/None strings

14 months agoContinued:
Roland Häder [Mon, 4 Sep 2023 07:54:14 +0000 (09:54 +0200)]
Continued:
- functions in module fba.helpers.tidyup are relatively "expensive", means they
  need a lot of CPU cycles
- let's avoid invoking them on empty string

14 months agoContinued:
Roland Häder [Sun, 3 Sep 2023 22:53:34 +0000 (00:53 +0200)]
Continued:
- added alias 'glitchcafe' for mastodon

14 months agoContinued:
Roland Häder [Sun, 3 Sep 2023 02:29:17 +0000 (04:29 +0200)]
Continued:
- renamed processing.domain() to processing.instance()
- renamed its parameter 'name' to 'blocked' (unified)

14 months agoContinued:
Roland Häder [Fri, 1 Sep 2023 23:43:35 +0000 (01:43 +0200)]
Continued:
- some messages are now info, one is debug

14 months agoContinued:
Roland Häder [Thu, 31 Aug 2023 06:00:52 +0000 (08:00 +0200)]
Continued:
- notice added about difference between total peer/block count

14 months agoContinued:
Roland Häder [Thu, 31 Aug 2023 05:53:16 +0000 (07:53 +0200)]
Continued:
- during fetching block lists or peers, filter out already blacklisted
  entries which will keep the list variable's memory usage smaller

14 months agoContinued:
Roland Häder [Wed, 30 Aug 2023 06:32:24 +0000 (08:32 +0200)]
Continued:
- formatted template block
- added average blocks

14 months agoContinued:
Roland Häder [Wed, 30 Aug 2023 01:31:34 +0000 (03:31 +0200)]
Continued:
- need to check blacklist/is_wanted()-check first before invoking
  blocks.is_instance_blocked()

14 months agoContinued:
Roland Häder [Tue, 29 Aug 2023 23:10:49 +0000 (01:10 +0200)]
Continued:
- removed a lot redundant messages, can be easier

14 months agoContinued:
Roland Häder [Tue, 29 Aug 2023 09:23:39 +0000 (11:23 +0200)]
Continued:
- added checks if instance is registered when update function are invoked
- some empty lines added

14 months agoContinued:
Roland Häder [Tue, 29 Aug 2023 06:46:05 +0000 (08:46 +0200)]
Continued:
- removed no longer used import

14 months agoAnother attempt to rewrite:
Roland Häder [Tue, 29 Aug 2023 05:52:42 +0000 (07:52 +0200)]
Another attempt to rewrite:
- don't update nodeinfo URL and detection mode to STATIC_CHECK while fetching
  blocks for Pleroma
- Pleroma has their block list exposed in that nodeinfo and not in separate API

14 months agoContinued:
Roland Häder [Mon, 28 Aug 2023 14:30:09 +0000 (16:30 +0200)]
Continued:
- some misskey instances may have no nodeinfo URL, e.g. they got only detected
  through APP_NAME method
- still they may provide a blocklist
- it is now rewritten that first a generic "/api/v1/instance/domain_blocks" is
  fetched and if it fails, a software-specific attempt is done

14 months agoContinued:
Roland Häder [Mon, 28 Aug 2023 12:11:50 +0000 (14:11 +0200)]
Continued:
- also need to break the loop when no 'data' element is being returned which
  might be the case when the server doesn't return a JSON

14 months agoContinued:
Roland Häder [Sun, 27 Aug 2023 14:27:54 +0000 (16:27 +0200)]
Continued:
- let's try it stricter: 200 OK is the only HTTP status code we accept
- log failing fetch of /instances from a Lemmy instance

14 months agoOps:
Roland Häder [Sun, 27 Aug 2023 06:02:01 +0000 (08:02 +0200)]
Ops:
- need to be negated
- moved LICENSE file to docs

14 months agoContinued:
Roland Häder [Sat, 26 Aug 2023 21:47:29 +0000 (23:47 +0200)]
Continued:
- if/elif/raise blocks should be separated from other logic
- tpzo fixed

14 months agoContinued:
Roland Häder [Sat, 26 Aug 2023 21:03:37 +0000 (23:03 +0200)]
Continued:
- warn about invalid SHA256 domain digests (aka. hashes)
- some mastodon instances maybe not include them at all?

14 months agoContinued:
Roland Häder [Sat, 26 Aug 2023 20:57:45 +0000 (22:57 +0200)]
Continued:
- fixed some pylint issues

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 23:31:52 +0000 (01:31 +0200)]
Continued:
- I hope this isn't to strict, some hosts return a "298 None" which the HTTP
  library doesn't see as failed (response.ok = False) but still doesn't return
  a JSON

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 19:41:11 +0000 (21:41 +0200)]
Continued:
- exclude NULL values for detection_mode

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 18:37:56 +0000 (20:37 +0200)]
Continued:
- half day for rechecking block lists was to fast
- also shortened API last access

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 15:53:11 +0000 (17:53 +0200)]
Continued:
- reformatted a bit

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 15:50:18 +0000 (17:50 +0200)]
Continued:
- fixed Shell script
- scoreboard for commands is total peers

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 01:02:21 +0000 (03:02 +0200)]
Continued:
- renamed update_data() to update()

14 months agoSorting order changed:
Roland Häder [Thu, 24 Aug 2023 22:07:51 +0000 (00:07 +0200)]
Sorting order changed:
- first total_peers/blocks descending
- then last_updated ascending

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 20:46:18 +0000 (22:46 +0200)]
Continued:
- debug log message added

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 20:35:33 +0000 (22:35 +0200)]
Continued:
- added alias 'revolver' as this is another one for pleroma

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 20:29:19 +0000 (22:29 +0200)]
Continued:
- fetch all data in /list view
- added total_peers and total_blocks to be exposed

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 19:12:14 +0000 (21:12 +0200)]
Continued:
- added parameter --software3 which searchinf for a file 'software.txt'
- you can generate this by running e.g.
  sqlite3 ./blocks.db "SELECT software FROM instances WHAT_EVER_PARAMETER;" > software.txt
- reset nodeinfo_url if software is now None
- always set complete URL, including domain

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 18:35:55 +0000 (20:35 +0200)]
Continued:
- renamed fetch_nodeinfo() to fetch() as it is already part of module nodeinfo
- added 3rd optional parameter to it, fetching of /.well-known/* isn't then
  required anymore and saves another request
- also the wanted URL can be directly used

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 16:59:38 +0000 (18:59 +0200)]
Continued:
- also order this list by total peers first, then row id

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 09:42:54 +0000 (11:42 +0200)]
Continued:
- also order by rowid descending

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 05:31:25 +0000 (07:31 +0200)]
Continued:
- sorted domains for fetch_blocks after how many total blocks have already been
  found

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 11:25:03 +0000 (13:25 +0200)]
Continued:
- neodb is providing at least /apt/v1/instance/peers

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 07:47:13 +0000 (09:47 +0200)]
Continued:
- strip spaces off

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 07:10:40 +0000 (09:10 +0200)]
Continued:
- added "dynamic alias" for software type 'gnusocial'

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 04:09:12 +0000 (06:09 +0200)]
Continued:
- added parameter --mode for updating node-info for detection mode
- this mode does only check outdated instances

14 months agoContinued:
Roland Häder [Tue, 22 Aug 2023 18:34:24 +0000 (20:34 +0200)]
Continued:
- added parameter --no-software which fetching instances with no software
  detected and not recently being checked
- the parameter --force is not re-checking recently entries. If you want this
  you need to use ./nodeinfo.sh --software --force but be kind to other
  webmasters!

14 months agoContinued:
Roland Häder [Fri, 18 Aug 2023 01:49:57 +0000 (03:49 +0200)]
Continued:
- some websites are so badly configured that they return a broken gzip body
- let's handle that exception here, too

14 months agoContinued:
Roland Häder [Thu, 17 Aug 2023 18:01:32 +0000 (20:01 +0200)]
Continued:
- wording fixed

14 months agoContinued:
Roland Häder [Thu, 17 Aug 2023 03:42:40 +0000 (05:42 +0200)]
Continued:
- ops, don't link None

14 months agoContinued:
Roland Häder [Wed, 16 Aug 2023 23:04:11 +0000 (01:04 +0200)]
Continued:
- added alias 'russkey' for 'misskey'

14 months agoContinued:
Roland Häder [Wed, 16 Aug 2023 22:56:19 +0000 (00:56 +0200)]
Continued:
- no, nope: validators.hostname() was a bad idea, it also let IP addresses and
  local host names in as well
- added command remove_invalid to remove those from database
- renamed recheck.sh -> nodeinfo.sh

14 months agoContinued:
Roland Häder [Wed, 16 Aug 2023 13:27:44 +0000 (15:27 +0200)]
Continued:
- chaos.social isn't part of oliphant, so it still requires being handled
  separately
- fetch software/origin from local database instead software from remote
  nodeinfo (saves some requests to their servers)

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 20:48:50 +0000 (22:48 +0200)]
Continued:
- links from infos.html view to list.html added

14 months agoFixed:
Roland Häder [Tue, 15 Aug 2023 20:24:41 +0000 (22:24 +0200)]
Fixed:
- I missed changing this one

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 20:09:54 +0000 (22:09 +0200)]
Continued:
- use validators.hostname() as these are host names

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 19:21:09 +0000 (21:21 +0200)]
Continued:
- fixed instance (host) names with two dots causing a UnicodeError

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 19:12:43 +0000 (21:12 +0200)]
Continued:
- need to skip recently but unknown instances (software is None but nodeinfo
  has recently being updated