]> git.mxchange.org Git - fba.git/log
fba.git
7 weeks agoContinued:
Roland Häder [Tue, 1 Oct 2024 16:52:05 +0000 (18:52 +0200)]
Continued:
- added function update() for "model" obfuscation (module name and table name
  must match as this is naming convention)
- added column 'last_used' to table 'obfuscation'

7 weeks agoContinued:
Roland Häder [Mon, 30 Sep 2024 19:23:42 +0000 (21:23 +0200)]
Continued:
- changed debug to info message as this was to quiet when for example the new
  instance isn't reachable (timeout)

7 weeks agoContinued:
Roland Häder [Sun, 29 Sep 2024 04:11:32 +0000 (06:11 +0200)]
Continued:
- added debug lines for function invocation
- change INFO into DEBUG message
- changed type to "any" (maybe it is limitable?)

2 months agoContinued:
Roland Häder [Sat, 21 Sep 2024 23:28:59 +0000 (01:28 +0200)]
Continued:
- more fixed for handing parked domains (200 OK or 302 but no real content)

2 months agoContinued:
Roland Häder [Sat, 21 Sep 2024 23:13:57 +0000 (01:13 +0200)]
Continued:
- some parked domains return "200 OK" for all requests. This is understandable
  in their view on things but it isn't standard-conform as 200 means document
  found and not some fake content being returned

2 months agoContinued:
Roland Häder [Thu, 19 Sep 2024 21:50:20 +0000 (23:50 +0200)]
Continued:
- removed obsolete code

2 months agoContinued:
Roland Häder [Thu, 19 Sep 2024 17:24:02 +0000 (19:24 +0200)]
Continued:
- an empty string is not the same as None, that's correct. So both need to be
  checked
- yes, first I check on false (missing keys, etc.) then true (positive outcome)

2 months agoContinued:
Roland Häder [Wed, 18 Sep 2024 22:37:32 +0000 (00:37 +0200)]
Continued:
- introduced network.fetch_json_rows() which fetches a JSON API response and
  returns a specifyable rows key (optional) to return as rows

2 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 17:17:41 +0000 (19:17 +0200)]
Continued:
- introduced domain_helper.is_tld_wanted() which checks if the top-level domain
  of a domain name is wanted

2 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 16:03:34 +0000 (18:03 +0200)]
Continued:
- skip invalid domains after checking them for obfuscation with asterisks or
  question marks

2 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 15:29:42 +0000 (17:29 +0200)]
Continued:
- rewrote chaos.social parser for their own documentation at meta.chaos.social

2 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 10:35:19 +0000 (12:35 +0200)]
Continued:
- instances.delete() now has an optional parameter 'column' which
  also needs validation on type and non-empty string
- some functions in same module didn't log optional/extra parameter 'column'

2 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 10:23:43 +0000 (12:23 +0200)]
Continued:
- these DELETE SQL statements have already been encapsulated

2 months agoContinued:
Roland Häder [Sat, 14 Sep 2024 18:43:56 +0000 (20:43 +0200)]
Continued:
- formatted longer SQL statment
- changed ordering of records

2 months agoContinued:
Roland Häder [Wed, 11 Sep 2024 23:29:52 +0000 (01:29 +0200)]
Continued:
- 'typeecho' sometimes provide a block list which has same API path as Fediverse
  instances provide, so let's add it as a source
- removed nodeinfo_url from SQL query when searching for software during
  fetch_blocks() command

2 months agoContinued:
Roland Häder [Sun, 8 Sep 2024 03:02:52 +0000 (05:02 +0200)]
Continued:
- added "logging" of obfuscated blocked domain/instance names
- if it has been deobfuscated, it will be removed

2 months agoContinued:
Roland Häder [Sun, 8 Sep 2024 00:16:26 +0000 (02:16 +0200)]
Continued:
- check if instance[host] is at least set and a non-empty string

2 months agoContinued:
Roland Häder [Sat, 7 Sep 2024 17:14:20 +0000 (19:14 +0200)]
Continued:
- when returned value from tidyup.domain() is an empty string or None then
  always log it as a debug message

2 months agoContinued:
Roland Häder [Wed, 4 Sep 2024 00:20:34 +0000 (02:20 +0200)]
Continued:
- added type-hint 'None' for all functions not returning a value, for example
  setters, add/update/delete functions

2 months agoContinued:
Roland Häder [Tue, 3 Sep 2024 23:07:37 +0000 (01:07 +0200)]
Continued:
- don't log an info after a "filter function"
- initialize variable 'search' as both if() blocks use the same code
- check variable 'domain' before 'char' as 'char' needs to be in a supported
  'domain' string

2 months agoContinued:
Roland Häder [Sat, 31 Aug 2024 18:12:20 +0000 (20:12 +0200)]
Continued:
- added another alias 'meowkey' for 'misskey'

2 months agoContinued:
Roland Häder [Sat, 24 Aug 2024 17:53:18 +0000 (19:53 +0200)]
Continued:
- Diaspora seem to provide an API for more peers, so let's add it for command
  fetch_instances()
- ... and yes, that long query isn't a good idea

3 months agoContinued:
Roland Häder [Fri, 23 Aug 2024 21:16:26 +0000 (23:16 +0200)]
Continued:
- added alias 'barkle' for 'misskey'

3 months agoContinued:
Roland Häder [Thu, 15 Aug 2024 19:35:57 +0000 (21:35 +0200)]
Continued:
- let also process.block() abort when an unwanted block_level is provided
- alias also "reject" to "rejected"

3 months agoContinued:
Roland Häder [Tue, 13 Aug 2024 21:22:01 +0000 (23:22 +0200)]
Continued:
- script is rejecting "reject" now in favor of "rejected" as there is
  "suspended" and "silenced" (past) already around
- Please run `$ sqlite3 blocks.db "UPDATE blocks SET block_level = 'rejected' WHERE block_level = 'reject';"`

3 months agoContinued:
Roland Häder [Wed, 7 Aug 2024 15:24:23 +0000 (17:24 +0200)]
Continued:
- update_nodeinfo() now accepts --force-recrawl and --force-update-none to have
  both "special" modes separately usable

3 months agoContinued:
Roland Häder [Wed, 7 Aug 2024 05:13:29 +0000 (07:13 +0200)]
Continued:
- --force is ambicious as now "other forces" will be added

3 months agoContinued:
Roland Häder [Wed, 31 Jul 2024 22:07:18 +0000 (00:07 +0200)]
Continued:
- added conditional links to 'added'/'updated' modes

3 months agoContinued:
Roland Häder [Mon, 29 Jul 2024 20:25:58 +0000 (22:25 +0200)]
Continued:
- update instance when error has happened (ops, missed this)
- also output warning when fields list is empty

3 months agoContinued:
Roland Häder [Mon, 29 Jul 2024 20:14:22 +0000 (22:14 +0200)]
Continued:
- added view for recently updated instances
- renamed 'recently' to 'added' for not confusing it

3 months agoContinued:
Roland Häder [Sat, 27 Jul 2024 21:51:00 +0000 (23:51 +0200)]
Continued:
- removed redundant utils.deobfuscate() invocation

3 months agoContinued:
Roland Häder [Sat, 27 Jul 2024 21:36:07 +0000 (23:36 +0200)]
Continued:
- reworked parts of finding out if blocker has blocked domains with
  placeholders like '*' or '?' in it
- improved some logger messages

3 months agoContinued:
Roland Häder [Thu, 25 Jul 2024 21:24:12 +0000 (23:24 +0200)]
Continued:
- added parameter --no-obfuscation to recheck records where blocked domains are
  not obfuscated before but maybe now and still have a software detected
- sort those recheck lists by last_blocked

4 months agoContinued:
Roland Häder [Sun, 21 Jul 2024 10:40:14 +0000 (12:40 +0200)]
Continued:
- added 3rd parameter for handling e.g. --force to fba.py or any other parameter
  fba.py handles
- more "debug" lines added

4 months agoContinued:
Roland Häder [Sat, 20 Jul 2024 18:29:23 +0000 (20:29 +0200)]
Continued:
- don't reduce counter for obfuscated domains, keep it at original value
- some more debug messages added

4 months agoContinued:
Roland Häder [Thu, 18 Jul 2024 22:16:38 +0000 (00:16 +0200)]
Continued:
- split off question mark (mostly comes from WordPress instances)

4 months agoContinued:
Roland Häder [Wed, 17 Jul 2024 18:07:17 +0000 (20:07 +0200)]
Continued:
- skip unset software (None)

4 months agoContinued:
Roland Häder [Sat, 13 Jul 2024 20:52:35 +0000 (22:52 +0200)]
Continued:
- removed never started feature to check local files containing blocked
  instances (and maybe reasons)

4 months agoContinued:
Roland Häder [Sat, 13 Jul 2024 20:49:32 +0000 (22:49 +0200)]
Continued:
- annotated more functions for caching
- initialized private _config "array" (dict)

4 months agoContinued:
Roland Häder [Sat, 13 Jul 2024 14:18:34 +0000 (16:18 +0200)]
Continued:
- skip invalid/unwanted domains (like .ip2/tld/onion ...)
- proper name logged

4 months agoContinued:
Roland Häder [Sat, 13 Jul 2024 13:12:30 +0000 (15:12 +0200)]
Continued:
- ops, forgot to add variable 'domain'

4 months agoContinued:
Roland Häder [Sat, 13 Jul 2024 09:21:44 +0000 (11:21 +0200)]
Continued:
- invalid domains should most be logged as warnings, not debug levels but
  sometimes it is okay
- allow RFC 2782 compliant domains (with underscore)

4 months agoContinued:
Roland Häder [Sat, 13 Jul 2024 09:19:07 +0000 (11:19 +0200)]
Continued:
- skip invalid domains (warning)
- skip .arpa/i2p/onion/tld (I2P might be allowed)

4 months agoContinued:
Roland Häder [Fri, 12 Jul 2024 09:53:38 +0000 (11:53 +0200)]
Continued:
- ops, renamed to much

4 months agoContinued:
Roland Häder [Wed, 10 Jul 2024 20:54:21 +0000 (22:54 +0200)]
Continued:
- a warning was to much here
- also need to check on other invalid TLDs, like .arpa, .tld and (if disabled)
  .onion/.i2p TLDs

4 months agoContinued:
Roland Häder [Wed, 10 Jul 2024 18:54:25 +0000 (20:54 +0200)]
Continued:
- allowed underscore in domains (RFC 2782)
- have to check domain validity before domain_helpers.encode_idna() is being
  invoked

4 months agoContinued:
Roland Häder [Wed, 10 Jul 2024 16:34:05 +0000 (18:34 +0200)]
Continued:
- introduced function domain_helper.encode_idna() which has centralized IDNA
  encoding

4 months agoContinued:
Roland Häder [Tue, 9 Jul 2024 23:09:45 +0000 (01:09 +0200)]
Continued:
- added 'wordpress' and 'oolong' as peer sources
- some APIs may return a dictionary (dict) for an instance entry

4 months agoContinued:
Roland Häder [Tue, 9 Jul 2024 22:34:12 +0000 (00:34 +0200)]
Continued:
- added "和歌" as alias for mastodon

4 months agoContinued:
Roland Häder [Tue, 9 Jul 2024 22:23:14 +0000 (00:23 +0200)]
Continued:
- added 'devtunnels.ms' to blacklist as this is only for testing/developing

4 months agoContinued:
Roland Häder [Mon, 8 Jul 2024 21:09:27 +0000 (23:09 +0200)]
Continued:
- removed redundant dashes

4 months agoContinued:
Roland Häder [Sun, 7 Jul 2024 21:18:20 +0000 (23:18 +0200)]
Continued:
- added 'kookie' as another alias for 'misskey'

4 months agoContinued:
Roland Häder [Sun, 7 Jul 2024 20:13:23 +0000 (22:13 +0200)]
Continued:
- moved args.single handling to proper level
- wrapped strings in Shell script
- explained all entries in .gitignore

4 months agoContinued:
Roland Häder [Mon, 1 Jul 2024 21:58:25 +0000 (23:58 +0200)]
Continued:
- stop question marks in domain.raise_on() function
- split off question mark
- added space for better readability
- fixed syntax error

4 months agoContinued:
Roland Häder [Mon, 1 Jul 2024 16:21:54 +0000 (18:21 +0200)]
Continued:
- WordPress is a peer list provider and can be utilized for more "peers" aka.
  instances
- URL from response object can be different than requested, it needs to be
  revalidated
- invalid redirect URLs are now logged with a level WARNING message

5 months agoContinued:
Roland Häder [Sun, 23 Jun 2024 11:28:52 +0000 (13:28 +0200)]
Continued:
- 'neodb' might appear in names such as 'testando neodb'

5 months agoContinued:
Roland Häder [Sat, 15 Jun 2024 13:11:40 +0000 (15:11 +0200)]
Continued:
- if the found digest isn't valid, then set it to None
- ignore (not skip) any invalid digests

5 months agoContinued:
Roland Häder [Fri, 14 Jun 2024 16:48:25 +0000 (18:48 +0200)]
Continued:
- fixed exception, need to check is_registered() here first

5 months agoContinued:
Roland Häder [Sun, 2 Jun 2024 14:53:24 +0000 (16:53 +0200)]
Continued:
- show run-mode of nodeinfo.sh
- fetch_instances can now (again) fetch new instances (not registered)

5 months agoContinued:
Roland Häder [Sat, 1 Jun 2024 19:21:39 +0000 (21:21 +0200)]
Continued:
- added another UTF-8 exception

5 months agoContinued:
Roland Häder [Sat, 25 May 2024 16:55:09 +0000 (18:55 +0200)]
Continued:
- added another exception

6 months agoContinued:
Roland Häder [Mon, 20 May 2024 19:23:31 +0000 (21:23 +0200)]
Continued:
- Hugo isn't distributed at all. I guess someone replaced a federated software
  with Hugo at the same domain

6 months agoContinued:
Roland Häder [Mon, 20 May 2024 14:36:51 +0000 (16:36 +0200)]
Continued:
- added 'biblioreads' as peer list provider

6 months agoContinued:
Roland Häder [Mon, 20 May 2024 12:49:13 +0000 (14:49 +0200)]
Continued:
- variable `response_url` is the whole URL, maybe stripped off from its anchor
  (#) part but it is NEVER a valid domain (ops)

6 months agoContinued:
Roland Häder [Mon, 20 May 2024 12:33:25 +0000 (14:33 +0200)]
Continued:
- some servers may return '<html something>' at the start (position 0),
- so if string.find() cannot find that string "<html", it will return -1 instead
  of zero

6 months agoContinued:
Roland Häder [Sat, 11 May 2024 19:36:59 +0000 (21:36 +0200)]
Continued:
- dict element 'domain' will always be there, no double-redundant checks

6 months agoContinued:
Roland Häder [Sat, 11 May 2024 19:30:26 +0000 (21:30 +0200)]
Continued:
- fetch_blocks_from_about() does return a 2-dimensional array, not a flat one

6 months agoContinued:
Roland Häder [Fri, 10 May 2024 20:56:07 +0000 (22:56 +0200)]
Continued:
- network aka. "universe" 'snac' added as peer source

6 months agoContinued:
Roland Häder [Sat, 4 May 2024 00:05:29 +0000 (02:05 +0200)]
Continued:
- addedd alias 'cutiekey' for 'misskey'

6 months agoContinued:
Roland Häder [Thu, 2 May 2024 01:23:12 +0000 (03:23 +0200)]
Continued:
- serveo.net blacklisted: 'Expose local servers to the internet'

6 months agoContinued:
Roland Häder [Tue, 30 Apr 2024 13:12:54 +0000 (15:12 +0200)]
Continued:
- nodeinfo_url should never be None/NULL here

7 months agoContinued:
Roland Häder [Tue, 23 Apr 2024 23:46:57 +0000 (01:46 +0200)]
Continued:
- blocked 'documentation.on.seirdy.one'
- removed old-lost blocklists
- added another CSV file for seirdy.one

7 months agoContinued:
Roland Häder [Sun, 21 Apr 2024 17:49:16 +0000 (19:49 +0200)]
Continued:
- rewrite all "*mammuthus*" to "mammuthus"

7 months agoContinued:
Roland Häder [Fri, 19 Apr 2024 23:41:39 +0000 (01:41 +0200)]
Continued:
- recheck_obfuscation now supports deleting unwanted domains from both tables
  'instances' and 'blocks'

7 months agoContinued:
Roland Häder [Fri, 19 Apr 2024 15:41:20 +0000 (17:41 +0200)]
Continued:
- moved some pre-sorting

7 months agoContinued:
Roland Häder [Thu, 18 Apr 2024 17:23:45 +0000 (19:23 +0200)]
Continued:
- added alias 'trashposs' for 'misskey'

7 months agoContinued:
Roland Häder [Wed, 17 Apr 2024 20:50:46 +0000 (22:50 +0200)]
Continued:
- added uncached blocks.get_reason()
- update block reason when current isn't set but updated reason is set

7 months agoContinued:
Roland Häder [Wed, 17 Apr 2024 19:38:13 +0000 (21:38 +0200)]
Continued:
- renamed local (overwritten) variable 'blocked' to 'cleared' so the array
  index can be found again

7 months agoContinued:
Roland Häder [Wed, 17 Apr 2024 18:10:06 +0000 (20:10 +0200)]
Continued:
- need to check if variable 'blocked' is None or an empty string first
- log size (len()) not whole array

7 months agoContinued:
Roland Häder [Tue, 16 Apr 2024 22:26:14 +0000 (00:26 +0200)]
Continued:
- added some missing instances.is_recent(foo) invocations
- use last_instance_fetch (default) where ever federation.fetch_instances() is
  been invoked
- wording fixed

7 months agoContinued:
Roland Häder [Tue, 16 Apr 2024 22:08:39 +0000 (00:08 +0200)]
Continued:
- added alias 'tanukey' for 'misskey'

7 months agoContinued:
Roland Häder [Tue, 16 Apr 2024 14:23:50 +0000 (16:23 +0200)]
Continued:
- blacklisted 'clients.your-server.de' and 2 others as this is a reverse-IP host
  name, but instances should have proper domain names

7 months agoContinued:
Roland Häder [Sun, 14 Apr 2024 09:54:15 +0000 (11:54 +0200)]
Continued:
- fixed 'TypeError: not enough arguments for format string'

7 months agoContinued:
Roland Häder [Wed, 10 Apr 2024 21:53:04 +0000 (23:53 +0200)]
Continued:
- cache blacklist function invocations

7 months agoContinued:
Roland Häder [Wed, 10 Apr 2024 21:38:28 +0000 (23:38 +0200)]
Continued:
- added 'toki' for peer lists, you can find it here:
  https://github.com/purifetchi/Toki

7 months agoContinued:
Roland Häder [Wed, 10 Apr 2024 09:56:11 +0000 (11:56 +0200)]
Continued:
- python3-aiofiles needs to be installed manually with Python 3.9
- Python 3.11 seem to install it automatically

7 months agoContinued:
Roland Häder [Tue, 9 Apr 2024 16:11:37 +0000 (18:11 +0200)]
Continued:
- added missing keyword (ops)

7 months agoContinued:
Roland Häder [Tue, 2 Apr 2024 01:39:32 +0000 (03:39 +0200)]
Continued:
- removed entries as the list has been deleted by oliphant + they provide
  their own version already

7 months agoContinued:
Roland Häder [Mon, 1 Apr 2024 22:56:35 +0000 (00:56 +0200)]
Continued:
- FBA now requires Python 3.11
- updated documentation accordingly
- removed fastapi and uvicorn requirements, your distribution must now provide
  it
- removed jinja2 and requests
- ignore entire venv/ directory, contains local paths

7 months agoContinued:
Roland Häder [Mon, 1 Apr 2024 03:12:17 +0000 (05:12 +0200)]
Continued:
- updated README file to reflect virtual environment for Python 3.11
- README file is a bit better

7 months agoContinued:
Roland Häder [Mon, 1 Apr 2024 03:01:19 +0000 (05:01 +0200)]
Continued:
- switched from global path to virtual-environment path

7 months agoContinued:
Roland Häder [Mon, 1 Apr 2024 02:49:23 +0000 (04:49 +0200)]
Continued:
- Python 3.11 needs a virtual environment
- so please add

source "${HOME}/fba/venv/bin/activate"

... to your ~/.bashrc

7 months agoContinued:
Roland Häder [Sun, 31 Mar 2024 21:05:47 +0000 (23:05 +0200)]
Continued:
- reduced duplicated code into a local variable 'response_url'
- need to recheck validity of 'response_url'

7 months agoContinued:
Roland Häder [Sun, 31 Mar 2024 08:00:43 +0000 (10:00 +0200)]
Continued:
- logged types (string or 'None' type)

7 months agoContinued:
Roland Häder [Sun, 31 Mar 2024 05:01:10 +0000 (07:01 +0200)]
Continued:
- handled ParserRejectedMarkup properly

7 months agoContinued:
Roland Häder [Sun, 31 Mar 2024 01:09:40 +0000 (03:09 +0200)]
Continued:
- added alias 'nexkey' for 'misskey'

8 months agoContinued:
Roland Häder [Wed, 20 Mar 2024 17:09:34 +0000 (18:09 +0100)]
Continued:
- added 'piefed' as possible source for block lists, still the network lacks
  API for instance list retrival

8 months agoContinued:
Roland Häder [Sun, 17 Mar 2024 00:07:51 +0000 (01:07 +0100)]
Continued:
- that Wiki page was listing "evil instances" and now explains the view of the
  owner of what FediBlock is

8 months agoContinued:
Roland Häder [Tue, 12 Mar 2024 12:50:39 +0000 (13:50 +0100)]
Continued:
- added 'hugo' as a source for more peers (only one so far)

8 months agoContinued:
Roland Häder [Tue, 12 Mar 2024 10:27:49 +0000 (11:27 +0100)]
Continued:
- added new aliases for pleroma
- moved pleroma aliases to "private" module variable